Jul 23 14:50:28.292928 ip-10-0-141-105 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 23 14:50:28.292944 ip-10-0-141-105 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 23 14:50:28.292954 ip-10-0-141-105 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 23 14:50:28.293268 ip-10-0-141-105 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 23 14:50:38.540065 ip-10-0-141-105 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 23 14:50:38.540081 ip-10-0-141-105 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot de2db79abfa84022b9bb397cb2ed65e9 --
Jul 23 14:52:37.200301 ip-10-0-141-105 systemd[1]: Starting Kubernetes Kubelet...
Jul 23 14:52:37.622057 ip-10-0-141-105 kubenswrapper[2560]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 23 14:52:37.622057 ip-10-0-141-105 kubenswrapper[2560]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 23 14:52:37.622057 ip-10-0-141-105 kubenswrapper[2560]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 23 14:52:37.622057 ip-10-0-141-105 kubenswrapper[2560]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 23 14:52:37.622057 ip-10-0-141-105 kubenswrapper[2560]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 23 14:52:37.624073 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.623922    2560 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 23 14:52:37.628356 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.628340    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 23 14:52:37.628356 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.628356    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 23 14:52:37.628419 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.628360    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 23 14:52:37.628419 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.628363    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 23 14:52:37.628419 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.628367    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 23 14:52:37.628419 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.628369    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 23 14:52:37.628419 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.628372    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 23 14:52:37.628419 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.628375    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 23 14:52:37.628419 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.628378    2560 feature_gate.go:328] unrecognized feature gate: Example2
Jul 23 14:52:37.628419 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.628380    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 23 14:52:37.628419 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.628383    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 23 14:52:37.628419 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.628385    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 23 14:52:37.628419 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.628388    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 23 14:52:37.628419 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.628398    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 23 14:52:37.628419 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.628401    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 23 14:52:37.628419 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.628403    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 23 14:52:37.628419 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.628406    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 23 14:52:37.628419 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.628408    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 23 14:52:37.628419 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.628412    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 23 14:52:37.628419 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.628416    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 23 14:52:37.628419 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.628419    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 23 14:52:37.628872 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.628422    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 23 14:52:37.628872 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.628425    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 23 14:52:37.628872 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.628428    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 23 14:52:37.628872 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.628432    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 23 14:52:37.628872 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.628436    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 23 14:52:37.628872 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.628439    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 23 14:52:37.628872 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.628443    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 23 14:52:37.628872 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.628445    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 23 14:52:37.628872 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.628448    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 23 14:52:37.628872 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.628450    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 23 14:52:37.628872 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.628453    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 23 14:52:37.628872 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.628455    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 23 14:52:37.628872 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.628458    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 23 14:52:37.628872 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.628460    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 23 14:52:37.628872 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.628463    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 23 14:52:37.628872 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.628465    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 23 14:52:37.628872 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.628467    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 23 14:52:37.628872 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.628470    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 23 14:52:37.628872 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.628473    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 23 14:52:37.628872 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.628475    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 23 14:52:37.629368 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.628478    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 23 14:52:37.629368 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.628480    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 23 14:52:37.629368 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.628483    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 23 14:52:37.629368 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.628486    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 23 14:52:37.629368 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.628488    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 23 14:52:37.629368 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.628490    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 23 14:52:37.629368 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.628493    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 23 14:52:37.629368 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.628495    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 23 14:52:37.629368 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.628499    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 23 14:52:37.629368 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.628501    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 23 14:52:37.629368 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.628503    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 23 14:52:37.629368 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.628505    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 23 14:52:37.629368 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.628508    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 23 14:52:37.629368 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.628511    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 23 14:52:37.629368 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.628514    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 23 14:52:37.629368 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.628517    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 23 14:52:37.629368 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.628519    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 23 14:52:37.629368 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.628522    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 23 14:52:37.629368 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.628525    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 23 14:52:37.629368 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.628527    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 23 14:52:37.629887 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.628530    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 23 14:52:37.629887 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.628532    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 23 14:52:37.629887 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.628535    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 23 14:52:37.629887 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.628537    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 23 14:52:37.629887 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.628540    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 23 14:52:37.629887 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.628542    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 23 14:52:37.629887 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.628546    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 23 14:52:37.629887 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.628548    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 23 14:52:37.629887 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.628551    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 23 14:52:37.629887 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.628553    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 23 14:52:37.629887 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.628555    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 23 14:52:37.629887 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.628558    2560 feature_gate.go:328] unrecognized feature gate: Example
Jul 23 14:52:37.629887 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.628561    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 23 14:52:37.629887 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.628563    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 23 14:52:37.629887 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.628566    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 23 14:52:37.629887 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.628568    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 23 14:52:37.629887 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.628571    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 23 14:52:37.629887 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.628573    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 23 14:52:37.629887 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.628576    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 23 14:52:37.630350 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.628578    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 23 14:52:37.630350 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.628580    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 23 14:52:37.630350 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.628583    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 23 14:52:37.630350 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.628586    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 23 14:52:37.630350 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.628588    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 23 14:52:37.630350 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.628591    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 23 14:52:37.630350 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.628594    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 23 14:52:37.630350 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.628597    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 23 14:52:37.630350 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.628599    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 23 14:52:37.630350 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.628602    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 23 14:52:37.630350 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.628604    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 23 14:52:37.630350 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.628607    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 23 14:52:37.630350 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.628609    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 23 14:52:37.630350 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.628612    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 23 14:52:37.630350 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.628614    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 23 14:52:37.630350 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.628616    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 23 14:52:37.630350 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.628619    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 23 14:52:37.630350 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.628624    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 23 14:52:37.630350 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.628627    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 23 14:52:37.630810 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.629370    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 23 14:52:37.630810 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.629378    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 23 14:52:37.630810 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.629382    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 23 14:52:37.630810 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.629385    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 23 14:52:37.630810 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.629388    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 23 14:52:37.630810 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.629391    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 23 14:52:37.630810 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.629394    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 23 14:52:37.630810 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.629397    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 23 14:52:37.630810 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.629400    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 23 14:52:37.630810 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.629403    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 23 14:52:37.630810 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.629405    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 23 14:52:37.630810 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.629408    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 23 14:52:37.630810 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.629411    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 23 14:52:37.630810 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.629413    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 23 14:52:37.630810 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.629416    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 23 14:52:37.630810 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.629418    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 23 14:52:37.630810 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.629421    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 23 14:52:37.630810 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.629423    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 23 14:52:37.630810 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.629426    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 23 14:52:37.630810 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.629430    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 23 14:52:37.631324 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.629434    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 23 14:52:37.631324 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.629436    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 23 14:52:37.631324 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.629439    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 23 14:52:37.631324 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.629441    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 23 14:52:37.631324 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.629444    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 23 14:52:37.631324 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.629446    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 23 14:52:37.631324 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.629450    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 23 14:52:37.631324 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.629452    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 23 14:52:37.631324 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.629455    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 23 14:52:37.631324 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.629457    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 23 14:52:37.631324 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.629460    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 23 14:52:37.631324 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.629463    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 23 14:52:37.631324 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.629465    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 23 14:52:37.631324 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.629468    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 23 14:52:37.631324 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.629470    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 23 14:52:37.631324 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.629473    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 23 14:52:37.631324 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.629475    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 23 14:52:37.631324 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.629478    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 23 14:52:37.631324 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.629480    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 23 14:52:37.631324 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.629483    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 23 14:52:37.631804 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.629486    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 23 14:52:37.631804 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.629488    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 23 14:52:37.631804 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.629491    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 23 14:52:37.631804 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.629493    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 23 14:52:37.631804 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.629497    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 23 14:52:37.631804 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.629501    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 23 14:52:37.631804 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.629503    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 23 14:52:37.631804 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.629506    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 23 14:52:37.631804 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.629508    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 23 14:52:37.631804 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.629511    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 23 14:52:37.631804 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.629513    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 23 14:52:37.631804 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.629517    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 23 14:52:37.631804 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.629519    2560 feature_gate.go:328] unrecognized feature gate: Example
Jul 23 14:52:37.631804 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.629522    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 23 14:52:37.631804 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.629524    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 23 14:52:37.631804 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.629527    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 23 14:52:37.631804 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.629529    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 23 14:52:37.631804 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.629532    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 23 14:52:37.631804 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.629534    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 23 14:52:37.632288 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.629537    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 23 14:52:37.632288 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.629539    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 23 14:52:37.632288 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.629542    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 23 14:52:37.632288 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.629544    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 23 14:52:37.632288 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.629550    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 23 14:52:37.632288 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.629553    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 23 14:52:37.632288 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.629556    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 23 14:52:37.632288 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.629559    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 23 14:52:37.632288 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.629561    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 23 14:52:37.632288 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.629564    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 23 14:52:37.632288 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.629567    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 23 14:52:37.632288 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.629569    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 23 14:52:37.632288 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.629572    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 23 14:52:37.632288 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.629576    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 23 14:52:37.632288 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.629579    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 23 14:52:37.632288 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.629583    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 23 14:52:37.632288 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.629586    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 23 14:52:37.632288 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.629588    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 23 14:52:37.632288 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.629591    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 23 14:52:37.632750 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.629594    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 23 14:52:37.632750 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.629596    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 23 14:52:37.632750 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.629599    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 23 14:52:37.632750 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.629601    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 23 14:52:37.632750 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.629603    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 23 14:52:37.632750 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.629607    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 23 14:52:37.632750 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.629609    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 23 14:52:37.632750 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.629612    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 23 14:52:37.632750 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.629614    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 23 14:52:37.632750 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.629617    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 23 14:52:37.632750 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.629619    2560 feature_gate.go:328] unrecognized feature gate: Example2
Jul 23 14:52:37.632750 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.629621    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 23 14:52:37.632750 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.629624    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 23 14:52:37.632750 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.629626    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 23 14:52:37.632750 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.629630    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 23 14:52:37.632750 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.629632    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 23 14:52:37.632750 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.629635    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 23 14:52:37.632750 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.629637    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 23 14:52:37.632750 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.629640    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 23 14:52:37.632750 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.629642    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 23 14:52:37.633251 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.629645    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 23 14:52:37.633251 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.630024    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 23 14:52:37.633251 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.630029    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 23 14:52:37.633251 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.630032    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 23 14:52:37.633251 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.630035    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 23 14:52:37.633251 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.630037    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 23 14:52:37.633251 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.630041    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 23 14:52:37.633251 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.630043    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 23 14:52:37.633251 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.630046    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 23 14:52:37.633251 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.630049    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 23 14:52:37.633251 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.630051    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 23 14:52:37.633251 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.630054    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 23 14:52:37.633251 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.630057    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 23 14:52:37.633251 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.630059    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 23 14:52:37.633251 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.630062    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 23 14:52:37.633251 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.630064    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 23 14:52:37.633251 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.630067    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 23 14:52:37.633251 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.630070    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 23 14:52:37.633251 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.630072    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 23 14:52:37.633251 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.630075    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 23 14:52:37.633732 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.630078    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 23 14:52:37.633732 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.630080    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 23 14:52:37.633732 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.630083    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 23 14:52:37.633732 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.630086    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 23 14:52:37.633732 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.630088    2560 feature_gate.go:328] unrecognized feature gate: Example
Jul 23 14:52:37.633732 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.630090    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 23 14:52:37.633732 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.630093    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 23 14:52:37.633732 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.630096    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 23 14:52:37.633732 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.630098    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 23 14:52:37.633732 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.630101    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 23 14:52:37.633732 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.630103    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 23 14:52:37.633732 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.630105    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 23 14:52:37.633732 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.630108    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 23 14:52:37.633732 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.630110    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 23 14:52:37.633732 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.630113    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 23 14:52:37.633732 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.630115    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 23 14:52:37.633732 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.630119    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 23 14:52:37.633732 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.630122    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 23 14:52:37.633732 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.630125    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 23 14:52:37.634201 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.630129    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 23 14:52:37.634201 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.630132    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 23 14:52:37.634201 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.630135    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 23 14:52:37.634201 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.630137    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 23 14:52:37.634201 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.630140    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 23 14:52:37.634201 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.630143    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 23 14:52:37.634201 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.630145    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 23 14:52:37.634201 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.630148    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 23 14:52:37.634201 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.630150    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 23 14:52:37.634201 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.630152    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 23 14:52:37.634201 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.630156    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 23 14:52:37.634201 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.630158    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 23 14:52:37.634201 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.630161    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 23 14:52:37.634201 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.630164    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 23 14:52:37.634201 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.630166    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 23 14:52:37.634201 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.630169    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 23 14:52:37.634201 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.630171    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 23 14:52:37.634201 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.630174    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 23 14:52:37.634201 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.630176    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 23 14:52:37.634201 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.630178    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 23 14:52:37.634740 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.630181    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 23 14:52:37.634740 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.630183    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 23 14:52:37.634740 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.630186    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 23 14:52:37.634740 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.630188    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 23 14:52:37.634740 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.630190    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 23 14:52:37.634740 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.630193    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 23 14:52:37.634740 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.630195    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 23 14:52:37.634740 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.630198    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 23 14:52:37.634740 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.630200    2560 feature_gate.go:328] unrecognized feature gate: Example2
Jul 23 14:52:37.634740 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.630203    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 23 14:52:37.634740 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.630205    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 23 14:52:37.634740 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.630208    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 23 14:52:37.634740 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.630210    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 23 14:52:37.634740 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.630214    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 23 14:52:37.634740 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.630218    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 23 14:52:37.634740 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.630223    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 23 14:52:37.634740 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.630227    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 23 14:52:37.634740 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.630230    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 23 14:52:37.634740 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.630234    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 23 14:52:37.635230 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.630237    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 23 14:52:37.635230 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.630240    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 23 14:52:37.635230 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.630243    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 23 14:52:37.635230 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.630247    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 23 14:52:37.635230 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.630250    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 23 14:52:37.635230 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.630253    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 23 14:52:37.635230 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.630255    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 23 14:52:37.635230 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.630258    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 23 14:52:37.635230 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.630260    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 23 14:52:37.635230 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.630263    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 23 14:52:37.635230 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.630265    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 23 14:52:37.635230 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.630268    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 23 14:52:37.635230 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.630270    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 23 14:52:37.635230 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.630273    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 23 14:52:37.635230 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.630275    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 23 14:52:37.635230 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.630278    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 23 14:52:37.635230 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.630280    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 23 14:52:37.635230 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.630282    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 23 14:52:37.635230 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.630285    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 23 14:52:37.635692 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.630288    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 23 14:52:37.635692 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.630290    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 23 14:52:37.635692 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.630293    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 23 14:52:37.635692 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.630360    2560 flags.go:64] FLAG: --address="0.0.0.0"
Jul 23 14:52:37.635692 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.630367    2560 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 23 14:52:37.635692 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.630373    2560 flags.go:64] FLAG: --anonymous-auth="true"
Jul 23 14:52:37.635692 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.630377    2560 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 23 14:52:37.635692 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.630381    2560 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 23 14:52:37.635692 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.630386    2560 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 23 14:52:37.635692 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.630391    2560 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 23 14:52:37.635692 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.630397    2560 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 23 14:52:37.635692 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.630401    2560 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 23 14:52:37.635692 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.630404    2560 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 23 14:52:37.635692 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.630407    2560 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 23 14:52:37.635692 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.630410    2560 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 23 14:52:37.635692 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.630413    2560 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 23 14:52:37.635692 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.630416    2560 flags.go:64] FLAG: --cgroup-root=""
Jul 23 14:52:37.635692 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.630419    2560 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 23 14:52:37.635692 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.630422    2560 flags.go:64] FLAG: --client-ca-file=""
Jul 23 14:52:37.635692 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.630425    2560 flags.go:64] FLAG: --cloud-provider="external"
Jul 23 14:52:37.635692 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.630428    2560 flags.go:64] FLAG: --cluster-dns="[]"
Jul 23 14:52:37.635692 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.630433    2560 flags.go:64] FLAG: --cluster-domain=""
Jul 23 14:52:37.635692 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.630436    2560 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 23 14:52:37.636266 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.630439    2560 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 23 14:52:37.636266 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.630442    2560 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 23 14:52:37.636266 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.630445    2560 flags.go:64] FLAG: --container-log-max-files="5"
Jul 23 14:52:37.636266 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.630449    2560 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 23 14:52:37.636266 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.630452    2560 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 23 14:52:37.636266 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.630455    2560 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 23 14:52:37.636266 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.630458    2560 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 23 14:52:37.636266 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.630460    2560 flags.go:64] FLAG: --contention-profiling="false"
Jul 23 14:52:37.636266 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.630463    2560 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 23 14:52:37.636266 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.630467    2560 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 23 14:52:37.636266 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.630470    2560 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 23 14:52:37.636266 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.630472    2560 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 23 14:52:37.636266 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.630476    2560 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 23 14:52:37.636266 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.630479    2560 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 23 14:52:37.636266 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.630482    2560 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 23 14:52:37.636266 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.630485    2560 flags.go:64] FLAG: --enable-load-reader="false"
Jul 23 14:52:37.636266 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.630488    2560 flags.go:64] FLAG: --enable-server="true"
Jul 23 14:52:37.636266 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.630491    2560 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 23 14:52:37.636266 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.630497    2560 flags.go:64] FLAG: --event-burst="100"
Jul 23 14:52:37.636266 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.630501    2560 flags.go:64] FLAG: --event-qps="50"
Jul 23 14:52:37.636266 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.630504    2560 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 23 14:52:37.636266 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.630507    2560 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 23 14:52:37.636266 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.630510    2560 flags.go:64] FLAG: --eviction-hard=""
Jul 23 14:52:37.636266 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.630514    2560 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 23 14:52:37.636840 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.630517    2560 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 23 14:52:37.636840 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.630520    2560 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 23 14:52:37.636840 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.630523    2560 flags.go:64] FLAG: --eviction-soft=""
Jul 23 14:52:37.636840 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.630526    2560 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 23 14:52:37.636840 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.630528    2560 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 23 14:52:37.636840 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.630531    2560 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 23 14:52:37.636840 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.630534    2560 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 23 14:52:37.636840 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.630537    2560 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 23 14:52:37.636840 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.630540    2560 flags.go:64] FLAG: --fail-swap-on="true"
Jul 23 14:52:37.636840 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.630544    2560 flags.go:64] FLAG: --feature-gates=""
Jul 23 14:52:37.636840 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.630547    2560 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 23 14:52:37.636840 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.630551    2560 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 23 14:52:37.636840 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.630554    2560 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 23 14:52:37.636840 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.630557    2560 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 23 14:52:37.636840 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.630560    2560 flags.go:64] FLAG: --healthz-port="10248"
Jul 23 14:52:37.636840 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.630563    2560 flags.go:64] FLAG: --help="false"
Jul 23 14:52:37.636840 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.630566    2560 flags.go:64] FLAG: --hostname-override="ip-10-0-141-105.ec2.internal"
Jul 23 14:52:37.636840 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.630569    2560 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 23 14:52:37.636840 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.630571    2560 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 23 14:52:37.636840 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.630574    2560 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 23 14:52:37.636840 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.630577    2560 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 23 14:52:37.636840 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.630581    2560 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 23 14:52:37.636840 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.630584    2560 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 23 14:52:37.636840 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.630587    2560 flags.go:64] FLAG: --image-service-endpoint=""
Jul 23 14:52:37.637520 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.630589    2560 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 23 14:52:37.637520 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.630592    2560 flags.go:64] FLAG: --kube-api-burst="100"
Jul 23 14:52:37.637520 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.630597    2560 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 23 14:52:37.637520 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.630604    2560 flags.go:64] FLAG: --kube-api-qps="50"
Jul 23 14:52:37.637520 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.630607    2560 flags.go:64] FLAG: --kube-reserved=""
Jul 23 14:52:37.637520 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.630610    2560 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 23 14:52:37.637520 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.630613    2560 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 23 14:52:37.637520 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.630616    2560 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 23 14:52:37.637520 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.630619    2560 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 23 14:52:37.637520 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.630622    2560 flags.go:64] FLAG: --lock-file=""
Jul 23 14:52:37.637520 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.630625    2560 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 23 14:52:37.637520 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.630628    2560 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 23 14:52:37.637520 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.630631    2560 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 23 14:52:37.637520 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.630636    2560 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 23 14:52:37.637520 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.630638    2560 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 23 14:52:37.637520 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.630641    2560 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 23 14:52:37.637520 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.630644    2560 flags.go:64] FLAG: --logging-format="text"
Jul 23 14:52:37.637520 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.630647    2560 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 23 14:52:37.637520 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.630651    2560 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 23 14:52:37.637520 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.630654    2560 flags.go:64] FLAG: --manifest-url=""
Jul 23 14:52:37.637520 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.630656    2560 flags.go:64] FLAG: --manifest-url-header=""
Jul 23 14:52:37.637520 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.630660    2560 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 23 14:52:37.637520 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.630663    2560 flags.go:64] FLAG: --max-open-files="1000000"
Jul 23 14:52:37.637520 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.630668    2560 flags.go:64] FLAG: --max-pods="110"
Jul 23 14:52:37.637520 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.630670    2560 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 23 14:52:37.638453 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.630673    2560 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 23 14:52:37.638453 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.630676    2560 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 23 14:52:37.638453 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.630679    2560 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 23 14:52:37.638453 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.630682    2560 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 23 14:52:37.638453 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.630685    2560 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 23 14:52:37.638453 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.630688    2560 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 23 14:52:37.638453 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.630695    2560 flags.go:64] FLAG: --node-status-max-images="50"
Jul 23 14:52:37.638453 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.630698    2560 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 23 14:52:37.638453 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.630701    2560 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 23 14:52:37.638453 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.630705    2560 flags.go:64] FLAG: --pod-cidr=""
Jul 23 14:52:37.638453 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.630709    2560 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0"
Jul 23 14:52:37.638453 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.630715    2560 flags.go:64] FLAG: --pod-manifest-path=""
Jul 23 14:52:37.638453 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.630718    2560 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 23 14:52:37.638453 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.630721    2560 flags.go:64] FLAG: --pods-per-core="0"
Jul 23 14:52:37.638453 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.630724    2560 flags.go:64] FLAG: --port="10250"
Jul 23 14:52:37.638453 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.630727    2560 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 23 14:52:37.638453 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.630730    2560 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0433b591c37c7a75c"
Jul 23 14:52:37.638453 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.630733    2560 flags.go:64] FLAG: --qos-reserved=""
Jul 23 14:52:37.638453 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.630736    2560 flags.go:64] FLAG: --read-only-port="10255"
Jul 23 14:52:37.638453 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.630739    2560 flags.go:64] FLAG: --register-node="true"
Jul 23 14:52:37.638453 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.630742    2560 flags.go:64] FLAG: --register-with-taints=""
Jul 23 14:52:37.638453 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.630750    2560 flags.go:64] FLAG: --registry-burst="10"
Jul 23 14:52:37.638453 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.630752    2560 flags.go:64] FLAG: --registry-qps="5"
Jul 23 14:52:37.638453 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.630755    2560 flags.go:64] FLAG: --reserved-cpus=""
Jul 23 14:52:37.639138 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.630758    2560 flags.go:64] FLAG: --reserved-memory=""
Jul 23 14:52:37.639138 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.630761    2560 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 23 14:52:37.639138 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.630764    2560 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 23 14:52:37.639138 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.630767    2560 flags.go:64] FLAG: --rotate-certificates="false"
Jul 23 14:52:37.639138 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.630770    2560 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 23 14:52:37.639138 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.630773    2560 flags.go:64] FLAG: --runonce="false"
Jul 23 14:52:37.639138 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.630776    2560 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 23 14:52:37.639138 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.630779    2560 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 23 14:52:37.639138 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.630782    2560 flags.go:64] FLAG: --seccomp-default="false"
Jul 23 14:52:37.639138 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.630784    2560 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 23 14:52:37.639138 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.630787    2560 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 23 14:52:37.639138 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.630790    2560 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 23 14:52:37.639138 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.630793    2560 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 23 14:52:37.639138 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.630796    2560 flags.go:64] FLAG: --storage-driver-password="root"
Jul 23 14:52:37.639138 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.630799    2560 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 23 14:52:37.639138 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.630802    2560 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 23 14:52:37.639138 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.630804    2560 flags.go:64] FLAG: --storage-driver-user="root"
Jul 23 14:52:37.639138 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.630809    2560 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 23 14:52:37.639138 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.630812    2560 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 23 14:52:37.639138 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.630818    2560 flags.go:64] FLAG: --system-cgroups=""
Jul 23 14:52:37.639138 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.630821    2560 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 23 14:52:37.639138 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.630827    2560 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 23 14:52:37.639138 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.630830    2560 flags.go:64] FLAG: --tls-cert-file=""
Jul 23 14:52:37.639138 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.630833    2560 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 23 14:52:37.639138 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.630838    2560 flags.go:64] FLAG: --tls-min-version=""
Jul 23 14:52:37.639747 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.630840    2560 flags.go:64] FLAG: --tls-private-key-file=""
Jul 23 14:52:37.639747 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.630843    2560 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 23 14:52:37.639747 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.630846    2560 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 23 14:52:37.639747 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.630849    2560 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 23 14:52:37.639747 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.630852    2560 flags.go:64] FLAG: --v="2"
Jul 23 14:52:37.639747 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.630856    2560 flags.go:64] FLAG: --version="false"
Jul 23 14:52:37.639747 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.630860    2560 flags.go:64] FLAG: --vmodule=""
Jul 23 14:52:37.639747 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.630863    2560 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 23 14:52:37.639747 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.630867    2560 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 23 14:52:37.639747 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.630992    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 23 14:52:37.639747 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.630996    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 23 14:52:37.639747 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.631000    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 23 14:52:37.639747 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.631003    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 23 14:52:37.639747 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.631006    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 23 14:52:37.639747 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.631010    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 23 14:52:37.639747 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.631012    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 23 14:52:37.639747 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.631015    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 23 14:52:37.639747 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.631018    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 23 14:52:37.639747 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.631021    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 23 14:52:37.639747 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.631023    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 23 14:52:37.639747 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.631026    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 23 14:52:37.640291 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.631028    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 23 14:52:37.640291 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.631031    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 23 14:52:37.640291 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.631033    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 23 14:52:37.640291 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.631036    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 23 14:52:37.640291 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.631039    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 23 14:52:37.640291 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.631042    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 23 14:52:37.640291 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.631045    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 23 14:52:37.640291 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.631048    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 23 14:52:37.640291 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.631050    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 23 14:52:37.640291 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.631053    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 23 14:52:37.640291 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.631055    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 23 14:52:37.640291 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.631057    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 23 14:52:37.640291 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.631060    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 23 14:52:37.640291 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.631062    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 23 14:52:37.640291 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.631065    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 23 14:52:37.640291 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.631067    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 23 14:52:37.640291 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.631070    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 23 14:52:37.640291 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.631072    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 23 14:52:37.640291 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.631074    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 23 14:52:37.640291 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.631077    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 23 14:52:37.640760 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.631080    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 23 14:52:37.640760 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.631082    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 23 14:52:37.640760 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.631084    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 23 14:52:37.640760 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.631087    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 23 14:52:37.640760 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.631089    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 23 14:52:37.640760 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.631091    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 23 14:52:37.640760 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.631094    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 23 14:52:37.640760 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.631097    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 23 14:52:37.640760 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.631099    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 23 14:52:37.640760 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.631101    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 23 14:52:37.640760 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.631104    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 23 14:52:37.640760 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.631106    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 23 14:52:37.640760 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.631109    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 23 14:52:37.640760 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.631111    2560 feature_gate.go:328] unrecognized feature gate: Example
Jul 23 14:52:37.640760 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.631113    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 23 14:52:37.640760 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.631116    2560 feature_gate.go:328] unrecognized feature gate: Example2
Jul 23 14:52:37.640760 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.631119    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 23 14:52:37.640760 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.631121    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 23 14:52:37.640760 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.631124    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 23 14:52:37.640760 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.631127    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 23 14:52:37.641252 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.631129    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 23 14:52:37.641252 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.631132    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 23 14:52:37.641252 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.631135    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 23 14:52:37.641252 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.631137    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 23 14:52:37.641252 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.631139    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 23 14:52:37.641252 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.631142    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 23 14:52:37.641252 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.631145    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 23 14:52:37.641252 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.631147    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 23 14:52:37.641252 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.631150    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 23 14:52:37.641252 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.631152    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 23 14:52:37.641252 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.631155    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 23 14:52:37.641252 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.631157    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 23 14:52:37.641252 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.631160    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 23 14:52:37.641252 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.631162    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 23 14:52:37.641252 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.631164    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 23 14:52:37.641252 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.631167    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 23 14:52:37.641252 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.631170    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 23 14:52:37.641252 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.631172    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 23 14:52:37.641252 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.631174    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 23 14:52:37.641252 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.631178    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 23 14:52:37.641736 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.631183    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 23 14:52:37.641736 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.631187    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 23 14:52:37.641736 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.631190    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 23 14:52:37.641736 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.631193    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 23 14:52:37.641736 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.631195    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 23 14:52:37.641736 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.631198    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 23 14:52:37.641736 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.631201    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 23 14:52:37.641736 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.631205    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 23 14:52:37.641736 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.631208    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 23 14:52:37.641736 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.631211    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 23 14:52:37.641736 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.631214    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 23 14:52:37.641736 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.631217    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 23 14:52:37.641736 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.631219    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 23 14:52:37.641736 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.631223    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 23 14:52:37.641736 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.631225    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 23 14:52:37.641736 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.631228    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 23 14:52:37.641736 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.631231    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 23 14:52:37.641736 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.631233    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 23 14:52:37.641736 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.631236    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 23 14:52:37.642294 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.631239    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 23 14:52:37.642294 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.631242    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 23 14:52:37.642294 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.631244    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 23 14:52:37.642294 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.631247    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 23 14:52:37.642294 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.631250    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 23 14:52:37.642294 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.631252    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 23 14:52:37.642294 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.631255    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 23 14:52:37.642294 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.631257    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 23 14:52:37.642294 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.631770    2560 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 23 14:52:37.642294 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.639737    2560 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 23 14:52:37.642294 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.639752    2560 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 23 14:52:37.642294 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.640712    2560 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 23 14:52:37.642294 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.640720    2560 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 23 14:52:37.642294 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.640779    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 23 14:52:37.642640 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.640785    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 23 14:52:37.642640 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.640789    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 23 14:52:37.642640 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.640792    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 23 14:52:37.642640 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.640796    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 23 14:52:37.642640 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.640799    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 23 14:52:37.642640 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.640802    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 23 14:52:37.642640 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.640804    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 23 14:52:37.642640 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.640807    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 23 14:52:37.642640 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.640810    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 23 14:52:37.642640 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.640812    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 23 14:52:37.642640 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.640815    2560 feature_gate.go:328] unrecognized feature gate: Example
Jul 23 14:52:37.642640 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.640818    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 23 14:52:37.642640 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.640820    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 23 14:52:37.642640 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.640823    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 23 14:52:37.642640 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.640825    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 23 14:52:37.642640 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.640828    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 23 14:52:37.642640 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.640831    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 23 14:52:37.642640 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.640833    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 23 14:52:37.642640 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.640836    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 23 14:52:37.643115 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.640838    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 23 14:52:37.643115 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.640842    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 23 14:52:37.643115 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.640844    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 23 14:52:37.643115 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.640847    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 23 14:52:37.643115 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.640849    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 23 14:52:37.643115 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.640852    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 23 14:52:37.643115 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.640855    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 23 14:52:37.643115 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.640857    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 23 14:52:37.643115 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.640860    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 23 14:52:37.643115 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.640863    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 23 14:52:37.643115 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.640866    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 23 14:52:37.643115 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.640868    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 23 14:52:37.643115 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.640871    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 23 14:52:37.643115 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.640873    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 23 14:52:37.643115 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.640876    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 23 14:52:37.643115 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.640878    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 23 14:52:37.643115 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.640881    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 23 14:52:37.643115 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.640883    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 23 14:52:37.643115 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.640886    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 23 14:52:37.643115 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.640889    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 23 14:52:37.643606 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.640891    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 23 14:52:37.643606 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.640893    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 23 14:52:37.643606 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.640895    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 23 14:52:37.643606 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.640898    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 23 14:52:37.643606 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.640901    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 23 14:52:37.643606 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.640904    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 23 14:52:37.643606 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.640908    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 23 14:52:37.643606 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.640917    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 23 14:52:37.643606 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.640921    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 23 14:52:37.643606 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.640923    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 23 14:52:37.643606 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.640926    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 23 14:52:37.643606 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.640930    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 23 14:52:37.643606 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.640934    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 23 14:52:37.643606 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.640937    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 23 14:52:37.643606 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.640940    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 23 14:52:37.643606 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.640943    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 23 14:52:37.643606 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.640945    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 23 14:52:37.643606 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.640948    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 23 14:52:37.643606 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.640971    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 23 14:52:37.644067 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.640975    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 23 14:52:37.644067 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.640978    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 23 14:52:37.644067 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.640982    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 23 14:52:37.644067 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.640985    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 23 14:52:37.644067 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.640987    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 23 14:52:37.644067 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.640990    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 23 14:52:37.644067 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.640993    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 23 14:52:37.644067 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.640995    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 23 14:52:37.644067 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.640998    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 23 14:52:37.644067 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.641001    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 23 14:52:37.644067 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.641004    2560 feature_gate.go:328] unrecognized feature gate: Example2
Jul 23 14:52:37.644067 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.641006    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 23 14:52:37.644067 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.641009    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 23 14:52:37.644067 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.641012    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 23 14:52:37.644067 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.641015    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 23 14:52:37.644067 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.641017    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 23 14:52:37.644067 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.641021    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 23 14:52:37.644067 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.641023    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 23 14:52:37.644067 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.641026    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 23 14:52:37.644571 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.641028    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 23 14:52:37.644571 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.641031    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 23 14:52:37.644571 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.641034    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 23 14:52:37.644571 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.641036    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 23 14:52:37.644571 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.641039    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 23 14:52:37.644571 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.641041    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 23 14:52:37.644571 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.641044    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 23 14:52:37.644571 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.641046    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 23 14:52:37.644571 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.641049    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 23 14:52:37.644571 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.641051    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 23 14:52:37.644571 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.641053    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 23 14:52:37.644571 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.641056    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 23 14:52:37.644571 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.641059    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 23 14:52:37.644571 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.641061    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 23 14:52:37.644571 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.641064    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 23 14:52:37.644571 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.641067    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 23 14:52:37.644571 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.641070    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 23 14:52:37.644571 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.641072    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 23 14:52:37.644571 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.641075    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 23 14:52:37.644571 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.641077    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 23 14:52:37.645074 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.641080    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 23 14:52:37.645074 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.641085    2560 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 23 14:52:37.645074 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.641205    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 23 14:52:37.645074 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.641210    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 23 14:52:37.645074 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.641214    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 23 14:52:37.645074 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.641217    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 23 14:52:37.645074 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.641219    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 23 14:52:37.645074 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.641222    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 23 14:52:37.645074 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.641225    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 23 14:52:37.645074 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.641228    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 23 14:52:37.645074 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.641231    2560 feature_gate.go:328] unrecognized feature gate: Example2
Jul 23 14:52:37.645074 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.641233    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 23 14:52:37.645074 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.641236    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 23 14:52:37.645074 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.641238    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 23 14:52:37.645074 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.641241    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 23 14:52:37.645439 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.641244    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 23 14:52:37.645439 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.641247    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 23 14:52:37.645439 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.641250    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 23 14:52:37.645439 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.641252    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 23 14:52:37.645439 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.641255    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 23 14:52:37.645439 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.641258    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 23 14:52:37.645439 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.641260    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 23 14:52:37.645439 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.641263    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 23 14:52:37.645439 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.641266    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 23 14:52:37.645439 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.641269    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 23 14:52:37.645439 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.641271    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 23 14:52:37.645439 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.641273    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 23 14:52:37.645439 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.641276    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 23 14:52:37.645439 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.641280    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 23 14:52:37.645439 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.641282    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 23 14:52:37.645439 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.641285    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 23 14:52:37.645439 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.641287    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 23 14:52:37.645439 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.641290    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 23 14:52:37.645439 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.641292    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 23 14:52:37.645439 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.641295    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 23 14:52:37.645912 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.641297    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 23 14:52:37.645912 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.641300    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 23 14:52:37.645912 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.641302    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 23 14:52:37.645912 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.641305    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 23 14:52:37.645912 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.641307    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 23 14:52:37.645912 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.641310    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 23 14:52:37.645912 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.641312    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 23 14:52:37.645912 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.641315    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 23 14:52:37.645912 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.641318    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 23 14:52:37.645912 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.641322    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 23 14:52:37.645912 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.641325    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 23 14:52:37.645912 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.641327    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 23 14:52:37.645912 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.641331    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 23 14:52:37.645912 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.641334    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 23 14:52:37.645912 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.641336    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 23 14:52:37.645912 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.641339    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 23 14:52:37.645912 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.641341    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 23 14:52:37.645912 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.641344    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 23 14:52:37.645912 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.641347    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 23 14:52:37.645912 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.641349    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 23 14:52:37.646412 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.641352    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 23 14:52:37.646412 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.641354    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 23 14:52:37.646412 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.641356    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 23 14:52:37.646412 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.641359    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 23 14:52:37.646412 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.641362    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 23 14:52:37.646412 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.641364    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 23 14:52:37.646412 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.641367    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 23 14:52:37.646412 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.641370    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 23 14:52:37.646412 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.641372    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 23 14:52:37.646412 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.641374    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 23 14:52:37.646412 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.641377    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 23 14:52:37.646412 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.641380    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 23 14:52:37.646412 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.641383    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 23 14:52:37.646412 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.641385    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 23 14:52:37.646412 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.641387    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 23 14:52:37.646412 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.641391    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 23 14:52:37.646412 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.641394    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 23 14:52:37.646412 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.641397    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 23 14:52:37.646412 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.641400    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 23 14:52:37.646870 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.641403    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 23 14:52:37.646870 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.641407    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 23 14:52:37.646870 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.641410    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 23 14:52:37.646870 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.641413    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 23 14:52:37.646870 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.641416    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 23 14:52:37.646870 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.641418    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 23 14:52:37.646870 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.641422    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 23 14:52:37.646870 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.641425    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 23 14:52:37.646870 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.641428    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 23 14:52:37.646870 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.641431    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 23 14:52:37.646870 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.641433    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 23 14:52:37.646870 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.641436    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 23 14:52:37.646870 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.641439    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 23 14:52:37.646870 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.641442    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 23 14:52:37.646870 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.641445    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 23 14:52:37.646870 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.641448    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 23 14:52:37.646870 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.641450    2560 feature_gate.go:328] unrecognized feature gate: Example
Jul 23 14:52:37.646870 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.641453    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 23 14:52:37.646870 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.641456    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 23 14:52:37.647360 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.641458    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 23 14:52:37.647360 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.641461    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 23 14:52:37.647360 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.641464    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 23 14:52:37.647360 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.641466    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 23 14:52:37.647360 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.641469    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 23 14:52:37.647360 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.641471    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 23 14:52:37.647360 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.641474    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 23 14:52:37.647360 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:37.641476    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 23 14:52:37.647360 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.641481    2560 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 23 14:52:37.647360 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.641588    2560 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 23 14:52:37.647360 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.643491    2560 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 23 14:52:37.647360 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.645073    2560 server.go:1019] "Starting client certificate rotation"
Jul 23 14:52:37.647360 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.645171    2560 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 23 14:52:37.647360 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.645204    2560 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 23 14:52:37.648036 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.648022    2560 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 23 14:52:37.649318 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.649304    2560 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 23 14:52:37.662730 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.662710    2560 log.go:25] "Validated CRI v1 runtime API"
Jul 23 14:52:37.664773 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.664756    2560 log.go:25] "Validated CRI v1 image API"
Jul 23 14:52:37.665596 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.665582    2560 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 23 14:52:37.667935 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.667917    2560 fs.go:135] Filesystem UUIDs: map[1f351b64-5475-45d3-8c0d-397866460188:/dev/nvme0n1p3 37fd5558-1fed-40ff-962f-ab7b476d70ee:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Jul 23 14:52:37.668001 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.667936    2560 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 23 14:52:37.674038 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.673903    2560 manager.go:216] Machine: {Timestamp:2026-07-23 14:52:37.671402198 +0000 UTC m=+0.350551001 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3094737 MemoryCapacity:32812167168 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec27ce9e961b0b18c1e03aba5d5d554e SystemUUID:ec27ce9e-961b-0b18-c1e0-3aba5d5d554e BootID:de2db79a-bfa8-4022-b9bb-397cb2ed65e9 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406081536 Type:vfs Inodes:4005391 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406085632 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:8a:17:f4:db:3f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:8a:17:f4:db:3f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:f6:53:e2:ad:6c:c0 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812167168 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 23 14:52:37.674038 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.674028    2560 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 23 14:52:37.674173 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.674107    2560 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 23 14:52:37.678982 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.678803    2560 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 23 14:52:37.679119 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.678986    2560 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-105.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 23 14:52:37.679672 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.679661    2560 topology_manager.go:138] "Creating topology manager with none policy"
Jul 23 14:52:37.679706 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.679676    2560 container_manager_linux.go:309] "Creating device plugin manager"
Jul 23 14:52:37.679706 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.679684    2560 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 23 14:52:37.680391 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.680380    2560 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 23 14:52:37.680632 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.680617    2560 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 23 14:52:37.681612 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.681599    2560 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 23 14:52:37.686103 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.686092    2560 state_mem.go:36] "Initialized new in-memory state store"
Jul 23 14:52:37.686225 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.686215    2560 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 23 14:52:37.686260 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.686245    2560 kubelet.go:486] "Attempting to sync node with API server"
Jul 23 14:52:37.686260 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.686256    2560 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 23 14:52:37.686311 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.686267    2560 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 23 14:52:37.686311 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.686275    2560 kubelet.go:398] "Adding apiserver pod source"
Jul 23 14:52:37.686311 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.686284    2560 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 23 14:52:37.686699 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.686685    2560 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 23 14:52:37.686758 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.686707    2560 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 23 14:52:37.689090 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.689074    2560 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 23 14:52:37.690357 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.690344    2560 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 23 14:52:37.690398 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.690375    2560 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 23 14:52:37.691199 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.691187    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 23 14:52:37.691242 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.691203    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 23 14:52:37.691242 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.691209    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 23 14:52:37.691242 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.691215    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 23 14:52:37.691242 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.691223    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 23 14:52:37.691242 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.691229    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 23 14:52:37.691242 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.691237    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 23 14:52:37.691242 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.691241    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 23 14:52:37.691426 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.691248    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 23 14:52:37.691426 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.691253    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 23 14:52:37.691426 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.691261    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 23 14:52:37.691426 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.691266    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 23 14:52:37.692388 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.692377    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 23 14:52:37.692388 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.692387    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 23 14:52:37.695540 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.695526    2560 server.go:1268] "Started kubelet"
Jul 23 14:52:37.695676 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.695612    2560 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 23 14:52:37.695771 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.695696    2560 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 23 14:52:37.696268 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.696221    2560 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 23 14:52:37.696370 ip-10-0-141-105 systemd[1]: Started Kubernetes Kubelet.
Jul 23 14:52:37.697169 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.697102    2560 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-141-105.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 23 14:52:37.697169 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:52:37.697121    2560 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 23 14:52:37.697273 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:52:37.697214    2560 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-141-105.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 23 14:52:37.698179 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.698167    2560 server.go:310] "Adding debug handlers to kubelet server"
Jul 23 14:52:37.699747 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.699729    2560 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 23 14:52:37.701784 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.701767    2560 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-nx26k"
Jul 23 14:52:37.702630 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:52:37.702610    2560 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 23 14:52:37.703185 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:52:37.702137    2560 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-105.ec2.internal.18c4f2d707094ebc  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-105.ec2.internal,UID:ip-10-0-141-105.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-141-105.ec2.internal,},FirstTimestamp:2026-07-23 14:52:37.695499964 +0000 UTC m=+0.374648765,LastTimestamp:2026-07-23 14:52:37.695499964 +0000 UTC m=+0.374648765,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-105.ec2.internal,}"
Jul 23 14:52:37.705157 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.705144    2560 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 23 14:52:37.705207 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.705199    2560 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 23 14:52:37.705337 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:52:37.705320    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-105.ec2.internal\" not found"
Jul 23 14:52:37.705766 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.705748    2560 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 23 14:52:37.705766 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.705751    2560 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 23 14:52:37.705888 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.705776    2560 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 23 14:52:37.705888 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.705875    2560 reconstruct.go:97] "Volume reconstruction finished"
Jul 23 14:52:37.705888 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.705883    2560 reconciler.go:29] "Reconciler: start to sync state"
Jul 23 14:52:37.706276 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.706263    2560 factory.go:153] Registering CRI-O factory
Jul 23 14:52:37.706332 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.706302    2560 factory.go:223] Registration of the crio container factory successfully
Jul 23 14:52:37.706484 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.706354    2560 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 23 14:52:37.706484 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.706364    2560 factory.go:55] Registering systemd factory
Jul 23 14:52:37.706484 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.706373    2560 factory.go:223] Registration of the systemd container factory successfully
Jul 23 14:52:37.706484 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.706393    2560 factory.go:103] Registering Raw factory
Jul 23 14:52:37.706484 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.706404    2560 manager.go:1196] Started watching for new ooms in manager
Jul 23 14:52:37.707046 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.707029    2560 manager.go:349] Starting recovery of all containers
Jul 23 14:52:37.710373 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:52:37.710341    2560 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-141-105.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 23 14:52:37.710501 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:52:37.710475    2560 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 23 14:52:37.711287 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.711193    2560 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-nx26k"
Jul 23 14:52:37.719005 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.718831    2560 manager.go:354] Recovery completed
Jul 23 14:52:37.722949 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.722936    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 23 14:52:37.724820 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.724801    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-105.ec2.internal" event="NodeHasSufficientMemory"
Jul 23 14:52:37.724877 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.724846    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-105.ec2.internal" event="NodeHasNoDiskPressure"
Jul 23 14:52:37.724877 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.724857    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-105.ec2.internal" event="NodeHasSufficientPID"
Jul 23 14:52:37.725346 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.725333    2560 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 23 14:52:37.725346 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.725345    2560 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 23 14:52:37.725439 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.725360    2560 state_mem.go:36] "Initialized new in-memory state store"
Jul 23 14:52:37.725439 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.725388    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 23 14:52:37.726029 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.726014    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-105.ec2.internal" event="NodeHasSufficientMemory"
Jul 23 14:52:37.726089 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.726045    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-105.ec2.internal" event="NodeHasNoDiskPressure"
Jul 23 14:52:37.726089 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.726057    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-105.ec2.internal" event="NodeHasSufficientPID"
Jul 23 14:52:37.726453 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.726442    2560 policy_none.go:49] "None policy: Start"
Jul 23 14:52:37.726487 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.726457    2560 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 23 14:52:37.726487 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.726467    2560 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 23 14:52:37.727688 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:52:37.727608    2560 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-105.ec2.internal.18c4f2d708c8a5e5  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-105.ec2.internal,UID:ip-10-0-141-105.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-141-105.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-141-105.ec2.internal,},FirstTimestamp:2026-07-23 14:52:37.724816869 +0000 UTC m=+0.403965673,LastTimestamp:2026-07-23 14:52:37.724816869 +0000 UTC m=+0.403965673,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-105.ec2.internal,}"
Jul 23 14:52:37.727771 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.727721    2560 policy_none.go:47] "Start"
Jul 23 14:52:37.765509 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.765494    2560 manager.go:337] "Starting Device Plugin manager"
Jul 23 14:52:37.765589 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:52:37.765523    2560 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 23 14:52:37.765589 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.765532    2560 server.go:87] "Starting device plugin registration server"
Jul 23 14:52:37.765768 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.765748    2560 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 23 14:52:37.765841 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.765769    2560 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 23 14:52:37.765888 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.765866    2560 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 23 14:52:37.765978 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.765947    2560 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 23 14:52:37.766036 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.765977    2560 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 23 14:52:37.766729 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:52:37.766707    2560 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 23 14:52:37.766805 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:52:37.766782    2560 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-105.ec2.internal\" not found"
Jul 23 14:52:37.866362 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.866324    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 23 14:52:37.867874 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.867320    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-105.ec2.internal" event="NodeHasSufficientMemory"
Jul 23 14:52:37.867874 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.867354    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-105.ec2.internal" event="NodeHasNoDiskPressure"
Jul 23 14:52:37.867874 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.867367    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-105.ec2.internal" event="NodeHasSufficientPID"
Jul 23 14:52:37.867874 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.867394    2560 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-141-105.ec2.internal"
Jul 23 14:52:37.873595 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.873547    2560 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-141-105.ec2.internal"
Jul 23 14:52:37.873595 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:52:37.873574    2560 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-141-105.ec2.internal\": node \"ip-10-0-141-105.ec2.internal\" not found"
Jul 23 14:52:37.883674 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.883645    2560 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 23 14:52:37.885011 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.884993    2560 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 23 14:52:37.885101 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.885031    2560 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 23 14:52:37.885101 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.885050    2560 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 23 14:52:37.885101 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:52:37.885079    2560 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 23 14:52:37.887020 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.887000    2560 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 23 14:52:37.889628 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:52:37.889606    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-105.ec2.internal\" not found"
Jul 23 14:52:37.986159 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.986126    2560 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-105.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-105.ec2.internal"]
Jul 23 14:52:37.986346 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.986219    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 23 14:52:37.987221 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.987206    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-105.ec2.internal" event="NodeHasSufficientMemory"
Jul 23 14:52:37.987318 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.987238    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-105.ec2.internal" event="NodeHasNoDiskPressure"
Jul 23 14:52:37.987318 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.987252    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-105.ec2.internal" event="NodeHasSufficientPID"
Jul 23 14:52:37.988574 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.988559    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 23 14:52:37.988710 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.988694    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-105.ec2.internal"
Jul 23 14:52:37.988759 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.988743    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 23 14:52:37.989315 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.989297    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-105.ec2.internal" event="NodeHasSufficientMemory"
Jul 23 14:52:37.989430 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.989329    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-105.ec2.internal" event="NodeHasNoDiskPressure"
Jul 23 14:52:37.989430 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.989297    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-105.ec2.internal" event="NodeHasSufficientMemory"
Jul 23 14:52:37.989430 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.989339    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-105.ec2.internal" event="NodeHasSufficientPID"
Jul 23 14:52:37.989430 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.989365    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-105.ec2.internal" event="NodeHasNoDiskPressure"
Jul 23 14:52:37.989430 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.989383    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-105.ec2.internal" event="NodeHasSufficientPID"
Jul 23 14:52:37.990061 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:52:37.990044    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-105.ec2.internal\" not found"
Jul 23 14:52:37.990575 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.990560    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-105.ec2.internal"
Jul 23 14:52:37.990646 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.990587    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 23 14:52:37.991244 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.991226    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-105.ec2.internal" event="NodeHasSufficientMemory"
Jul 23 14:52:37.991336 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.991254    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-105.ec2.internal" event="NodeHasNoDiskPressure"
Jul 23 14:52:37.991336 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:37.991264    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-105.ec2.internal" event="NodeHasSufficientPID"
Jul 23 14:52:38.007513 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:38.007492    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/666bf80ba1e1983d2d3dd851bc8ab130-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-105.ec2.internal\" (UID: \"666bf80ba1e1983d2d3dd851bc8ab130\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-105.ec2.internal"
Jul 23 14:52:38.007590 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:38.007517    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/666bf80ba1e1983d2d3dd851bc8ab130-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-105.ec2.internal\" (UID: \"666bf80ba1e1983d2d3dd851bc8ab130\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-105.ec2.internal"
Jul 23 14:52:38.007590 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:38.007533    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3d37fd728dd468b7ee1c820042e75b2d-config\") pod \"kube-apiserver-proxy-ip-10-0-141-105.ec2.internal\" (UID: \"3d37fd728dd468b7ee1c820042e75b2d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-105.ec2.internal"
Jul 23 14:52:38.017648 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:52:38.017627    2560 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-105.ec2.internal\" not found" node="ip-10-0-141-105.ec2.internal"
Jul 23 14:52:38.022053 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:52:38.022039    2560 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-105.ec2.internal\" not found" node="ip-10-0-141-105.ec2.internal"
Jul 23 14:52:38.091001 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:52:38.090977    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-105.ec2.internal\" not found"
Jul 23 14:52:38.108358 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:38.108333    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/666bf80ba1e1983d2d3dd851bc8ab130-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-105.ec2.internal\" (UID: \"666bf80ba1e1983d2d3dd851bc8ab130\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-105.ec2.internal"
Jul 23 14:52:38.108429 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:38.108361    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/666bf80ba1e1983d2d3dd851bc8ab130-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-105.ec2.internal\" (UID: \"666bf80ba1e1983d2d3dd851bc8ab130\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-105.ec2.internal"
Jul 23 14:52:38.108429 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:38.108379    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3d37fd728dd468b7ee1c820042e75b2d-config\") pod \"kube-apiserver-proxy-ip-10-0-141-105.ec2.internal\" (UID: \"3d37fd728dd468b7ee1c820042e75b2d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-105.ec2.internal"
Jul 23 14:52:38.108429 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:38.108422    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/666bf80ba1e1983d2d3dd851bc8ab130-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-105.ec2.internal\" (UID: \"666bf80ba1e1983d2d3dd851bc8ab130\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-105.ec2.internal"
Jul 23 14:52:38.108530 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:38.108421    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/666bf80ba1e1983d2d3dd851bc8ab130-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-105.ec2.internal\" (UID: \"666bf80ba1e1983d2d3dd851bc8ab130\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-105.ec2.internal"
Jul 23 14:52:38.108530 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:38.108449    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3d37fd728dd468b7ee1c820042e75b2d-config\") pod \"kube-apiserver-proxy-ip-10-0-141-105.ec2.internal\" (UID: \"3d37fd728dd468b7ee1c820042e75b2d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-105.ec2.internal"
Jul 23 14:52:38.191856 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:52:38.191792    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-105.ec2.internal\" not found"
Jul 23 14:52:38.292338 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:52:38.292304    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-105.ec2.internal\" not found"
Jul 23 14:52:38.319875 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:38.319851    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-105.ec2.internal"
Jul 23 14:52:38.326527 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:38.326512    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-105.ec2.internal"
Jul 23 14:52:38.393228 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:52:38.393194    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-105.ec2.internal\" not found"
Jul 23 14:52:39.665074 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:52:39.664999    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-105.ec2.internal\" not found"
Jul 23 14:52:39.765529 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:52:39.765493    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-105.ec2.internal\" not found"
Jul 23 14:52:39.816258 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:39.816212    2560 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 23 14:52:39.816846 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:39.816453    2560 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 23 14:52:39.865654 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:52:39.865613    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-105.ec2.internal\" not found"
Jul 23 14:52:39.890637 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:39.890165    2560 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-22 14:47:38 +0000 UTC" deadline="2028-02-17 02:23:44.379749983 +0000 UTC"
Jul 23 14:52:39.890637 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:39.890189    2560 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13763h31m4.489563354s"
Jul 23 14:52:39.966200 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:52:39.966131    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-105.ec2.internal\" not found"
Jul 23 14:52:39.983333 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:39.983291    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d37fd728dd468b7ee1c820042e75b2d.slice/crio-0b86a509e36f2127689c0207eb9f57ff37db417690793a42c449c444a20d22e7 WatchSource:0}: Error finding container 0b86a509e36f2127689c0207eb9f57ff37db417690793a42c449c444a20d22e7: Status 404 returned error can't find the container with id 0b86a509e36f2127689c0207eb9f57ff37db417690793a42c449c444a20d22e7
Jul 23 14:52:39.987497 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:39.987472    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 23 14:52:40.019172 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.019135    2560 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 23 14:52:40.034525 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:40.034503    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod666bf80ba1e1983d2d3dd851bc8ab130.slice/crio-4855aca11512df8b46ab450d8bd1cc7d2e12a98581d22dac6608d06589af955c WatchSource:0}: Error finding container 4855aca11512df8b46ab450d8bd1cc7d2e12a98581d22dac6608d06589af955c: Status 404 returned error can't find the container with id 4855aca11512df8b46ab450d8bd1cc7d2e12a98581d22dac6608d06589af955c
Jul 23 14:52:40.059531 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.059472    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-105.ec2.internal" event={"ID":"666bf80ba1e1983d2d3dd851bc8ab130","Type":"ContainerStarted","Data":"4855aca11512df8b46ab450d8bd1cc7d2e12a98581d22dac6608d06589af955c"}
Jul 23 14:52:40.060476 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.060453    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-105.ec2.internal" event={"ID":"3d37fd728dd468b7ee1c820042e75b2d","Type":"ContainerStarted","Data":"0b86a509e36f2127689c0207eb9f57ff37db417690793a42c449c444a20d22e7"}
Jul 23 14:52:40.076642 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.076617    2560 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-105.ec2.internal"
Jul 23 14:52:40.087983 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.087958    2560 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 23 14:52:40.089932 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.089915    2560 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-105.ec2.internal"
Jul 23 14:52:40.102531 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.102507    2560 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 23 14:52:40.161297 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.161271    2560 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 23 14:52:40.208837 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.208809    2560 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 23 14:52:40.859909 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.859377    2560 apiserver.go:52] "Watching apiserver"
Jul 23 14:52:40.868833 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.868804    2560 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 23 14:52:40.869321 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.869288    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-7sqtc","openshift-network-operator/iptables-alerter-gd6w9","openshift-cluster-node-tuning-operator/tuned-rmbz7","openshift-image-registry/node-ca-lz8qd","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-105.ec2.internal","openshift-multus/multus-additional-cni-plugins-q79jt","openshift-multus/network-metrics-daemon-hvw8v","openshift-ovn-kubernetes/ovnkube-node-z4fvn","kube-system/global-pull-secret-syncer-hvsnb","kube-system/konnectivity-agent-ns8qd","kube-system/kube-apiserver-proxy-ip-10-0-141-105.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9qk7h","openshift-dns/node-resolver-vfsh6","openshift-multus/multus-68mqt"]
Jul 23 14:52:40.871008 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.870987    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hvw8v"
Jul 23 14:52:40.871119 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:52:40.871065    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hvw8v" podUID="413d2464-7258-4246-85bb-ac9f03976ee5"
Jul 23 14:52:40.873315 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.872905    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-z4fvn"
Jul 23 14:52:40.874465 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.874441    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rmbz7"
Jul 23 14:52:40.875848 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.875826    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-lz8qd"
Jul 23 14:52:40.878444 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.876836    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-4m8j4\""
Jul 23 14:52:40.878444 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.877545    2560 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 23 14:52:40.878444 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.877779    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 23 14:52:40.878444 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.877972    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 23 14:52:40.878444 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.878200    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 23 14:52:40.878444 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.878304    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 23 14:52:40.878813 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.878509    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 23 14:52:40.878813 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.878695    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 23 14:52:40.878910 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.878877    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 23 14:52:40.879067 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.879043    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7sqtc"
Jul 23 14:52:40.879165 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.879119    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-9rbqv\""
Jul 23 14:52:40.879165 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:52:40.879121    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7sqtc" podUID="3370857a-3ffb-4603-bcad-503b48c10fd6"
Jul 23 14:52:40.879333 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.879317    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 23 14:52:40.879386 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.879331    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 23 14:52:40.879533 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.879519    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-q79jt"
Jul 23 14:52:40.880587 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.880139    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 23 14:52:40.880587 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.880213    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 23 14:52:40.880781 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.880766    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hvsnb"
Jul 23 14:52:40.880844 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:52:40.880825    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hvsnb" podUID="527775ad-7c13-4732-b824-4302ad79dcf4"
Jul 23 14:52:40.882223 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.882202    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-vfsh6"
Jul 23 14:52:40.883896 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.883872    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 23 14:52:40.884455 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.884437    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 23 14:52:40.884551 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.884474    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 23 14:52:40.884820 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.884806    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 23 14:52:40.887286 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.885195    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 23 14:52:40.887286 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.885527    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-gd6w9"
Jul 23 14:52:40.887286 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.885635    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-jdfqr\""
Jul 23 14:52:40.887286 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.886282    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-npssl\""
Jul 23 14:52:40.887286 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.886302    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-x8f5b\""
Jul 23 14:52:40.888214 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.888189    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e0c66f26-06fe-4273-ba29-fa20ce89b873-ovnkube-config\") pod \"ovnkube-node-z4fvn\" (UID: \"e0c66f26-06fe-4273-ba29-fa20ce89b873\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4fvn"
Jul 23 14:52:40.888340 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.888242    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/37ed7912-d183-41e3-994b-561a8815f7d5-run\") pod \"tuned-rmbz7\" (UID: \"37ed7912-d183-41e3-994b-561a8815f7d5\") " pod="openshift-cluster-node-tuning-operator/tuned-rmbz7"
Jul 23 14:52:40.888340 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.888270    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/37ed7912-d183-41e3-994b-561a8815f7d5-var-lib-kubelet\") pod \"tuned-rmbz7\" (UID: \"37ed7912-d183-41e3-994b-561a8815f7d5\") " pod="openshift-cluster-node-tuning-operator/tuned-rmbz7"
Jul 23 14:52:40.888340 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.888337    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2hgq\" (UniqueName: \"kubernetes.io/projected/9a2b87d6-d3dd-41b8-96f8-2e7a0be09325-kube-api-access-r2hgq\") pod \"node-ca-lz8qd\" (UID: \"9a2b87d6-d3dd-41b8-96f8-2e7a0be09325\") " pod="openshift-image-registry/node-ca-lz8qd"
Jul 23 14:52:40.888535 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.888352    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e0c66f26-06fe-4273-ba29-fa20ce89b873-host-run-netns\") pod \"ovnkube-node-z4fvn\" (UID: \"e0c66f26-06fe-4273-ba29-fa20ce89b873\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4fvn"
Jul 23 14:52:40.888535 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.888404    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e0c66f26-06fe-4273-ba29-fa20ce89b873-var-lib-openvswitch\") pod \"ovnkube-node-z4fvn\" (UID: \"e0c66f26-06fe-4273-ba29-fa20ce89b873\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4fvn"
Jul 23 14:52:40.888535 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.888472    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e0c66f26-06fe-4273-ba29-fa20ce89b873-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-z4fvn\" (UID: \"e0c66f26-06fe-4273-ba29-fa20ce89b873\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4fvn"
Jul 23 14:52:40.888535 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.888498    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/37ed7912-d183-41e3-994b-561a8815f7d5-etc-tuned\") pod \"tuned-rmbz7\" (UID: \"37ed7912-d183-41e3-994b-561a8815f7d5\") " pod="openshift-cluster-node-tuning-operator/tuned-rmbz7"
Jul 23 14:52:40.888764 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.888547    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9a2b87d6-d3dd-41b8-96f8-2e7a0be09325-serviceca\") pod \"node-ca-lz8qd\" (UID: \"9a2b87d6-d3dd-41b8-96f8-2e7a0be09325\") " pod="openshift-image-registry/node-ca-lz8qd"
Jul 23 14:52:40.888907 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.888888    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e0c66f26-06fe-4273-ba29-fa20ce89b873-run-systemd\") pod \"ovnkube-node-z4fvn\" (UID: \"e0c66f26-06fe-4273-ba29-fa20ce89b873\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4fvn"
Jul 23 14:52:40.889043 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.889022    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-ns8qd"
Jul 23 14:52:40.889262 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.889026    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e0c66f26-06fe-4273-ba29-fa20ce89b873-run-ovn\") pod \"ovnkube-node-z4fvn\" (UID: \"e0c66f26-06fe-4273-ba29-fa20ce89b873\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4fvn"
Jul 23 14:52:40.889315 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.889272    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/37ed7912-d183-41e3-994b-561a8815f7d5-etc-kubernetes\") pod \"tuned-rmbz7\" (UID: \"37ed7912-d183-41e3-994b-561a8815f7d5\") " pod="openshift-cluster-node-tuning-operator/tuned-rmbz7"
Jul 23 14:52:40.889315 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.889304    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5gs7\" (UniqueName: \"kubernetes.io/projected/37ed7912-d183-41e3-994b-561a8815f7d5-kube-api-access-q5gs7\") pod \"tuned-rmbz7\" (UID: \"37ed7912-d183-41e3-994b-561a8815f7d5\") " pod="openshift-cluster-node-tuning-operator/tuned-rmbz7"
Jul 23 14:52:40.889375 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.889333    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e0c66f26-06fe-4273-ba29-fa20ce89b873-run-openvswitch\") pod \"ovnkube-node-z4fvn\" (UID: \"e0c66f26-06fe-4273-ba29-fa20ce89b873\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4fvn"
Jul 23 14:52:40.889406 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.889372    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e0c66f26-06fe-4273-ba29-fa20ce89b873-host-cni-netd\") pod \"ovnkube-node-z4fvn\" (UID: \"e0c66f26-06fe-4273-ba29-fa20ce89b873\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4fvn"
Jul 23 14:52:40.889438 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.889413    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r266g\" (UniqueName: \"kubernetes.io/projected/e0c66f26-06fe-4273-ba29-fa20ce89b873-kube-api-access-r266g\") pod \"ovnkube-node-z4fvn\" (UID: \"e0c66f26-06fe-4273-ba29-fa20ce89b873\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4fvn"
Jul 23 14:52:40.889438 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.889430    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxfps\" (UniqueName: \"kubernetes.io/projected/413d2464-7258-4246-85bb-ac9f03976ee5-kube-api-access-gxfps\") pod \"network-metrics-daemon-hvw8v\" (UID: \"413d2464-7258-4246-85bb-ac9f03976ee5\") " pod="openshift-multus/network-metrics-daemon-hvw8v"
Jul 23 14:52:40.889532 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.889449    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e0c66f26-06fe-4273-ba29-fa20ce89b873-host-slash\") pod \"ovnkube-node-z4fvn\" (UID: \"e0c66f26-06fe-4273-ba29-fa20ce89b873\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4fvn"
Jul 23 14:52:40.889532 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.889501    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e0c66f26-06fe-4273-ba29-fa20ce89b873-etc-openvswitch\") pod \"ovnkube-node-z4fvn\" (UID: \"e0c66f26-06fe-4273-ba29-fa20ce89b873\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4fvn"
Jul 23 14:52:40.889624 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.889532    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e0c66f26-06fe-4273-ba29-fa20ce89b873-host-run-ovn-kubernetes\") pod \"ovnkube-node-z4fvn\" (UID: \"e0c66f26-06fe-4273-ba29-fa20ce89b873\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4fvn"
Jul 23 14:52:40.889624 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.889561    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e0c66f26-06fe-4273-ba29-fa20ce89b873-env-overrides\") pod \"ovnkube-node-z4fvn\" (UID: \"e0c66f26-06fe-4273-ba29-fa20ce89b873\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4fvn"
Jul 23 14:52:40.889624 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.889585    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e0c66f26-06fe-4273-ba29-fa20ce89b873-ovn-node-metrics-cert\") pod \"ovnkube-node-z4fvn\" (UID: \"e0c66f26-06fe-4273-ba29-fa20ce89b873\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4fvn"
Jul 23 14:52:40.889624 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.889611    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/37ed7912-d183-41e3-994b-561a8815f7d5-etc-sysconfig\") pod \"tuned-rmbz7\" (UID: \"37ed7912-d183-41e3-994b-561a8815f7d5\") " pod="openshift-cluster-node-tuning-operator/tuned-rmbz7"
Jul 23 14:52:40.889926 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.889636    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 23 14:52:40.889926 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.889640    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/37ed7912-d183-41e3-994b-561a8815f7d5-etc-systemd\") pod \"tuned-rmbz7\" (UID: \"37ed7912-d183-41e3-994b-561a8815f7d5\") " pod="openshift-cluster-node-tuning-operator/tuned-rmbz7"
Jul 23 14:52:40.889926 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.889677    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/37ed7912-d183-41e3-994b-561a8815f7d5-tmp\") pod \"tuned-rmbz7\" (UID: \"37ed7912-d183-41e3-994b-561a8815f7d5\") " pod="openshift-cluster-node-tuning-operator/tuned-rmbz7"
Jul 23 14:52:40.889926 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.889699    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9a2b87d6-d3dd-41b8-96f8-2e7a0be09325-host\") pod \"node-ca-lz8qd\" (UID: \"9a2b87d6-d3dd-41b8-96f8-2e7a0be09325\") " pod="openshift-image-registry/node-ca-lz8qd"
Jul 23 14:52:40.889926 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.889736    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e0c66f26-06fe-4273-ba29-fa20ce89b873-host-kubelet\") pod \"ovnkube-node-z4fvn\" (UID: \"e0c66f26-06fe-4273-ba29-fa20ce89b873\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4fvn"
Jul 23 14:52:40.889926 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.889754    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e0c66f26-06fe-4273-ba29-fa20ce89b873-systemd-units\") pod \"ovnkube-node-z4fvn\" (UID: \"e0c66f26-06fe-4273-ba29-fa20ce89b873\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4fvn"
Jul 23 14:52:40.889926 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.889771    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/37ed7912-d183-41e3-994b-561a8815f7d5-etc-modprobe-d\") pod \"tuned-rmbz7\" (UID: \"37ed7912-d183-41e3-994b-561a8815f7d5\") " pod="openshift-cluster-node-tuning-operator/tuned-rmbz7"
Jul 23 14:52:40.889926 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.889807    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/37ed7912-d183-41e3-994b-561a8815f7d5-etc-sysctl-d\") pod \"tuned-rmbz7\" (UID: \"37ed7912-d183-41e3-994b-561a8815f7d5\") " pod="openshift-cluster-node-tuning-operator/tuned-rmbz7"
Jul 23 14:52:40.889926 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.889823    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/37ed7912-d183-41e3-994b-561a8815f7d5-etc-sysctl-conf\") pod \"tuned-rmbz7\" (UID: \"37ed7912-d183-41e3-994b-561a8815f7d5\") " pod="openshift-cluster-node-tuning-operator/tuned-rmbz7"
Jul 23 14:52:40.889926 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.889841    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/37ed7912-d183-41e3-994b-561a8815f7d5-host\") pod \"tuned-rmbz7\" (UID: \"37ed7912-d183-41e3-994b-561a8815f7d5\") " pod="openshift-cluster-node-tuning-operator/tuned-rmbz7"
Jul 23 14:52:40.889926 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.889849    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-q94wp\""
Jul 23 14:52:40.889926 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.889860    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcqwm\" (UniqueName: \"kubernetes.io/projected/3370857a-3ffb-4603-bcad-503b48c10fd6-kube-api-access-fcqwm\") pod \"network-check-target-7sqtc\" (UID: \"3370857a-3ffb-4603-bcad-503b48c10fd6\") " pod="openshift-network-diagnostics/network-check-target-7sqtc"
Jul 23 14:52:40.889926 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.889876    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/413d2464-7258-4246-85bb-ac9f03976ee5-metrics-certs\") pod \"network-metrics-daemon-hvw8v\" (UID: \"413d2464-7258-4246-85bb-ac9f03976ee5\") " pod="openshift-multus/network-metrics-daemon-hvw8v"
Jul 23 14:52:40.889926 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.889906    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e0c66f26-06fe-4273-ba29-fa20ce89b873-node-log\") pod \"ovnkube-node-z4fvn\" (UID: \"e0c66f26-06fe-4273-ba29-fa20ce89b873\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4fvn"
Jul 23 14:52:40.889926 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.889934    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e0c66f26-06fe-4273-ba29-fa20ce89b873-log-socket\") pod \"ovnkube-node-z4fvn\" (UID: \"e0c66f26-06fe-4273-ba29-fa20ce89b873\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4fvn"
Jul 23 14:52:40.890604 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.889958    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e0c66f26-06fe-4273-ba29-fa20ce89b873-host-cni-bin\") pod \"ovnkube-node-z4fvn\" (UID: \"e0c66f26-06fe-4273-ba29-fa20ce89b873\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4fvn"
Jul 23 14:52:40.890604 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.889982    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e0c66f26-06fe-4273-ba29-fa20ce89b873-ovnkube-script-lib\") pod \"ovnkube-node-z4fvn\" (UID: \"e0c66f26-06fe-4273-ba29-fa20ce89b873\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4fvn"
Jul 23 14:52:40.890604 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.889990    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 23 14:52:40.890604 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.890005    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/37ed7912-d183-41e3-994b-561a8815f7d5-sys\") pod \"tuned-rmbz7\" (UID: \"37ed7912-d183-41e3-994b-561a8815f7d5\") " pod="openshift-cluster-node-tuning-operator/tuned-rmbz7"
Jul 23 14:52:40.890604 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.890029    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/37ed7912-d183-41e3-994b-561a8815f7d5-lib-modules\") pod \"tuned-rmbz7\" (UID: \"37ed7912-d183-41e3-994b-561a8815f7d5\") " pod="openshift-cluster-node-tuning-operator/tuned-rmbz7"
Jul 23 14:52:40.890604 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.890366    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 23 14:52:40.890876 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.890756    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 23 14:52:40.891512 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.891479    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-68mqt"
Jul 23 14:52:40.891756 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.891738    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 23 14:52:40.892727 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.892710    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-gbr2c\""
Jul 23 14:52:40.892814 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.892746    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 23 14:52:40.892875 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.892832    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 23 14:52:40.893884 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.893864    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9qk7h"
Jul 23 14:52:40.894421 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.894405    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 23 14:52:40.895255 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.895215    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-d2tcz\""
Jul 23 14:52:40.896732 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.896715    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 23 14:52:40.898413 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.898383    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 23 14:52:40.898413 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.898402    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-wd46w\""
Jul 23 14:52:40.898665 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.898649    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 23 14:52:40.901383 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.901362    2560 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 23 14:52:40.929746 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.929713    2560 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-xzlrv"
Jul 23 14:52:40.939370 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.939348    2560 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-xzlrv"
Jul 23 14:52:40.978413 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.978384    2560 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 23 14:52:40.990673 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.990648    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e0c66f26-06fe-4273-ba29-fa20ce89b873-ovnkube-script-lib\") pod \"ovnkube-node-z4fvn\" (UID: \"e0c66f26-06fe-4273-ba29-fa20ce89b873\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4fvn"
Jul 23 14:52:40.990815 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.990679    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/37ed7912-d183-41e3-994b-561a8815f7d5-sys\") pod \"tuned-rmbz7\" (UID: \"37ed7912-d183-41e3-994b-561a8815f7d5\") " pod="openshift-cluster-node-tuning-operator/tuned-rmbz7"
Jul 23 14:52:40.990815 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.990746    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/37ed7912-d183-41e3-994b-561a8815f7d5-sys\") pod \"tuned-rmbz7\" (UID: \"37ed7912-d183-41e3-994b-561a8815f7d5\") " pod="openshift-cluster-node-tuning-operator/tuned-rmbz7"
Jul 23 14:52:40.990815 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.990801    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/37ed7912-d183-41e3-994b-561a8815f7d5-lib-modules\") pod \"tuned-rmbz7\" (UID: \"37ed7912-d183-41e3-994b-561a8815f7d5\") " pod="openshift-cluster-node-tuning-operator/tuned-rmbz7"
Jul 23 14:52:40.990951 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.990838    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcldm\" (UniqueName: \"kubernetes.io/projected/95f95dc6-4b74-428b-8d51-d7975d5ed0e2-kube-api-access-xcldm\") pod \"aws-ebs-csi-driver-node-9qk7h\" (UID: \"95f95dc6-4b74-428b-8d51-d7975d5ed0e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9qk7h"
Jul 23 14:52:40.990951 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.990866    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/527775ad-7c13-4732-b824-4302ad79dcf4-kubelet-config\") pod \"global-pull-secret-syncer-hvsnb\" (UID: \"527775ad-7c13-4732-b824-4302ad79dcf4\") " pod="kube-system/global-pull-secret-syncer-hvsnb"
Jul 23 14:52:40.990951 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.990895    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e0c66f26-06fe-4273-ba29-fa20ce89b873-ovnkube-config\") pod \"ovnkube-node-z4fvn\" (UID: \"e0c66f26-06fe-4273-ba29-fa20ce89b873\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4fvn"
Jul 23 14:52:40.990951 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.990919    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/37ed7912-d183-41e3-994b-561a8815f7d5-run\") pod \"tuned-rmbz7\" (UID: \"37ed7912-d183-41e3-994b-561a8815f7d5\") " pod="openshift-cluster-node-tuning-operator/tuned-rmbz7"
Jul 23 14:52:40.990951 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.990942    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2hgq\" (UniqueName: \"kubernetes.io/projected/9a2b87d6-d3dd-41b8-96f8-2e7a0be09325-kube-api-access-r2hgq\") pod \"node-ca-lz8qd\" (UID: \"9a2b87d6-d3dd-41b8-96f8-2e7a0be09325\") " pod="openshift-image-registry/node-ca-lz8qd"
Jul 23 14:52:40.991123 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.990967    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c4471b3d-89b5-46d0-b16d-734e52f04801-host-slash\") pod \"iptables-alerter-gd6w9\" (UID: \"c4471b3d-89b5-46d0-b16d-734e52f04801\") " pod="openshift-network-operator/iptables-alerter-gd6w9"
Jul 23 14:52:40.991123 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.990991    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bc81888a-6094-4c2e-b881-6b9cd628e6c1-cni-binary-copy\") pod \"multus-68mqt\" (UID: \"bc81888a-6094-4c2e-b881-6b9cd628e6c1\") " pod="openshift-multus/multus-68mqt"
Jul 23 14:52:40.991123 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.991014    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bc81888a-6094-4c2e-b881-6b9cd628e6c1-multus-daemon-config\") pod \"multus-68mqt\" (UID: \"bc81888a-6094-4c2e-b881-6b9cd628e6c1\") " pod="openshift-multus/multus-68mqt"
Jul 23 14:52:40.991123 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.991037    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8d4t\" (UniqueName: \"kubernetes.io/projected/bc81888a-6094-4c2e-b881-6b9cd628e6c1-kube-api-access-x8d4t\") pod \"multus-68mqt\" (UID: \"bc81888a-6094-4c2e-b881-6b9cd628e6c1\") " pod="openshift-multus/multus-68mqt"
Jul 23 14:52:40.991123 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.991061    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6fbaa452-6ab5-4370-ae18-fb4290ce9c10-cni-binary-copy\") pod \"multus-additional-cni-plugins-q79jt\" (UID: \"6fbaa452-6ab5-4370-ae18-fb4290ce9c10\") " pod="openshift-multus/multus-additional-cni-plugins-q79jt"
Jul 23 14:52:40.991123 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.991088    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e0c66f26-06fe-4273-ba29-fa20ce89b873-host-run-netns\") pod \"ovnkube-node-z4fvn\" (UID: \"e0c66f26-06fe-4273-ba29-fa20ce89b873\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4fvn"
Jul 23 14:52:40.991123 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.991094    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/37ed7912-d183-41e3-994b-561a8815f7d5-run\") pod \"tuned-rmbz7\" (UID: \"37ed7912-d183-41e3-994b-561a8815f7d5\") " pod="openshift-cluster-node-tuning-operator/tuned-rmbz7"
Jul 23 14:52:40.991123 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.991112    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6nwq\" (UniqueName: \"kubernetes.io/projected/d9b3550d-e82f-4934-b093-486563f240fb-kube-api-access-d6nwq\") pod \"node-resolver-vfsh6\" (UID: \"d9b3550d-e82f-4934-b093-486563f240fb\") " pod="openshift-dns/node-resolver-vfsh6"
Jul 23 14:52:40.991500 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.991138    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bc81888a-6094-4c2e-b881-6b9cd628e6c1-host-var-lib-cni-bin\") pod \"multus-68mqt\" (UID: \"bc81888a-6094-4c2e-b881-6b9cd628e6c1\") " pod="openshift-multus/multus-68mqt"
Jul 23 14:52:40.991500 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.991172    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6fbaa452-6ab5-4370-ae18-fb4290ce9c10-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-q79jt\" (UID: \"6fbaa452-6ab5-4370-ae18-fb4290ce9c10\") " pod="openshift-multus/multus-additional-cni-plugins-q79jt"
Jul 23 14:52:40.991500 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.990993    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/37ed7912-d183-41e3-994b-561a8815f7d5-lib-modules\") pod \"tuned-rmbz7\" (UID: \"37ed7912-d183-41e3-994b-561a8815f7d5\") " pod="openshift-cluster-node-tuning-operator/tuned-rmbz7"
Jul 23 14:52:40.991500 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.991270    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e0c66f26-06fe-4273-ba29-fa20ce89b873-ovnkube-script-lib\") pod \"ovnkube-node-z4fvn\" (UID: \"e0c66f26-06fe-4273-ba29-fa20ce89b873\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4fvn"
Jul 23 14:52:40.991500 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.991343    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e0c66f26-06fe-4273-ba29-fa20ce89b873-host-run-netns\") pod \"ovnkube-node-z4fvn\" (UID: \"e0c66f26-06fe-4273-ba29-fa20ce89b873\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4fvn"
Jul 23 14:52:40.991500 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.991377    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e0c66f26-06fe-4273-ba29-fa20ce89b873-run-systemd\") pod \"ovnkube-node-z4fvn\" (UID: \"e0c66f26-06fe-4273-ba29-fa20ce89b873\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4fvn"
Jul 23 14:52:40.991500 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.991403    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/95f95dc6-4b74-428b-8d51-d7975d5ed0e2-socket-dir\") pod \"aws-ebs-csi-driver-node-9qk7h\" (UID: \"95f95dc6-4b74-428b-8d51-d7975d5ed0e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9qk7h"
Jul 23 14:52:40.991500 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.991441    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6fbaa452-6ab5-4370-ae18-fb4290ce9c10-cnibin\") pod \"multus-additional-cni-plugins-q79jt\" (UID: \"6fbaa452-6ab5-4370-ae18-fb4290ce9c10\") " pod="openshift-multus/multus-additional-cni-plugins-q79jt"
Jul 23 14:52:40.991500 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.991439    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e0c66f26-06fe-4273-ba29-fa20ce89b873-run-systemd\") pod \"ovnkube-node-z4fvn\" (UID: \"e0c66f26-06fe-4273-ba29-fa20ce89b873\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4fvn"
Jul 23 14:52:40.991500 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.991476    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6fbaa452-6ab5-4370-ae18-fb4290ce9c10-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-q79jt\" (UID: \"6fbaa452-6ab5-4370-ae18-fb4290ce9c10\") " pod="openshift-multus/multus-additional-cni-plugins-q79jt"
Jul 23 14:52:40.991915 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.991512    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ht5cd\" (UniqueName: \"kubernetes.io/projected/6fbaa452-6ab5-4370-ae18-fb4290ce9c10-kube-api-access-ht5cd\") pod \"multus-additional-cni-plugins-q79jt\" (UID: \"6fbaa452-6ab5-4370-ae18-fb4290ce9c10\") " pod="openshift-multus/multus-additional-cni-plugins-q79jt"
Jul 23 14:52:40.991915 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.991542    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e0c66f26-06fe-4273-ba29-fa20ce89b873-run-openvswitch\") pod \"ovnkube-node-z4fvn\" (UID: \"e0c66f26-06fe-4273-ba29-fa20ce89b873\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4fvn"
Jul 23 14:52:40.991915 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.991568    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e0c66f26-06fe-4273-ba29-fa20ce89b873-host-cni-netd\") pod \"ovnkube-node-z4fvn\" (UID: \"e0c66f26-06fe-4273-ba29-fa20ce89b873\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4fvn"
Jul 23 14:52:40.991915 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.991594    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e0c66f26-06fe-4273-ba29-fa20ce89b873-run-openvswitch\") pod \"ovnkube-node-z4fvn\" (UID: \"e0c66f26-06fe-4273-ba29-fa20ce89b873\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4fvn"
Jul 23 14:52:40.991915 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.991613    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bc81888a-6094-4c2e-b881-6b9cd628e6c1-multus-cni-dir\") pod \"multus-68mqt\" (UID: \"bc81888a-6094-4c2e-b881-6b9cd628e6c1\") " pod="openshift-multus/multus-68mqt"
Jul 23 14:52:40.991915 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.991646    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b3df911b-3c63-441b-97fa-8d431bf3fb95-konnectivity-ca\") pod \"konnectivity-agent-ns8qd\" (UID: \"b3df911b-3c63-441b-97fa-8d431bf3fb95\") " pod="kube-system/konnectivity-agent-ns8qd"
Jul 23 14:52:40.991915 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.991657    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e0c66f26-06fe-4273-ba29-fa20ce89b873-ovnkube-config\") pod \"ovnkube-node-z4fvn\" (UID: \"e0c66f26-06fe-4273-ba29-fa20ce89b873\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4fvn"
Jul 23 14:52:40.991915 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.991667    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e0c66f26-06fe-4273-ba29-fa20ce89b873-host-cni-netd\") pod \"ovnkube-node-z4fvn\" (UID: \"e0c66f26-06fe-4273-ba29-fa20ce89b873\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4fvn"
Jul 23 14:52:40.991915 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.991692    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r266g\" (UniqueName: \"kubernetes.io/projected/e0c66f26-06fe-4273-ba29-fa20ce89b873-kube-api-access-r266g\") pod \"ovnkube-node-z4fvn\" (UID: \"e0c66f26-06fe-4273-ba29-fa20ce89b873\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4fvn"
Jul 23 14:52:40.991915 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.991725    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxfps\" (UniqueName: \"kubernetes.io/projected/413d2464-7258-4246-85bb-ac9f03976ee5-kube-api-access-gxfps\") pod \"network-metrics-daemon-hvw8v\" (UID: \"413d2464-7258-4246-85bb-ac9f03976ee5\") " pod="openshift-multus/network-metrics-daemon-hvw8v"
Jul 23 14:52:40.991915 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.991857    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e0c66f26-06fe-4273-ba29-fa20ce89b873-etc-openvswitch\") pod \"ovnkube-node-z4fvn\" (UID: \"e0c66f26-06fe-4273-ba29-fa20ce89b873\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4fvn"
Jul 23 14:52:40.992310 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.991963    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e0c66f26-06fe-4273-ba29-fa20ce89b873-host-run-ovn-kubernetes\") pod \"ovnkube-node-z4fvn\" (UID: \"e0c66f26-06fe-4273-ba29-fa20ce89b873\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4fvn"
Jul 23 14:52:40.992310 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.991976    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e0c66f26-06fe-4273-ba29-fa20ce89b873-etc-openvswitch\") pod \"ovnkube-node-z4fvn\" (UID: \"e0c66f26-06fe-4273-ba29-fa20ce89b873\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4fvn"
Jul 23 14:52:40.992310 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.992008    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/95f95dc6-4b74-428b-8d51-d7975d5ed0e2-registration-dir\") pod \"aws-ebs-csi-driver-node-9qk7h\" (UID: \"95f95dc6-4b74-428b-8d51-d7975d5ed0e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9qk7h"
Jul 23 14:52:40.992310 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.992033    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c4471b3d-89b5-46d0-b16d-734e52f04801-iptables-alerter-script\") pod \"iptables-alerter-gd6w9\" (UID: \"c4471b3d-89b5-46d0-b16d-734e52f04801\") " pod="openshift-network-operator/iptables-alerter-gd6w9"
Jul 23 14:52:40.992310 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.992064    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/37ed7912-d183-41e3-994b-561a8815f7d5-etc-systemd\") pod \"tuned-rmbz7\" (UID: \"37ed7912-d183-41e3-994b-561a8815f7d5\") " pod="openshift-cluster-node-tuning-operator/tuned-rmbz7"
Jul 23 14:52:40.992310 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.992087    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/37ed7912-d183-41e3-994b-561a8815f7d5-tmp\") pod \"tuned-rmbz7\" (UID: \"37ed7912-d183-41e3-994b-561a8815f7d5\") " pod="openshift-cluster-node-tuning-operator/tuned-rmbz7"
Jul 23 14:52:40.992310 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.992087    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e0c66f26-06fe-4273-ba29-fa20ce89b873-host-run-ovn-kubernetes\") pod \"ovnkube-node-z4fvn\" (UID: \"e0c66f26-06fe-4273-ba29-fa20ce89b873\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4fvn"
Jul 23 14:52:40.992310 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.992119    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9a2b87d6-d3dd-41b8-96f8-2e7a0be09325-host\") pod \"node-ca-lz8qd\" (UID: \"9a2b87d6-d3dd-41b8-96f8-2e7a0be09325\") " pod="openshift-image-registry/node-ca-lz8qd"
Jul 23 14:52:40.992310 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.992144    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/95f95dc6-4b74-428b-8d51-d7975d5ed0e2-device-dir\") pod \"aws-ebs-csi-driver-node-9qk7h\" (UID: \"95f95dc6-4b74-428b-8d51-d7975d5ed0e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9qk7h"
Jul 23 14:52:40.992310 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.992159    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/37ed7912-d183-41e3-994b-561a8815f7d5-etc-systemd\") pod \"tuned-rmbz7\" (UID: \"37ed7912-d183-41e3-994b-561a8815f7d5\") " pod="openshift-cluster-node-tuning-operator/tuned-rmbz7"
Jul 23 14:52:40.992310 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.992168    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6fbaa452-6ab5-4370-ae18-fb4290ce9c10-os-release\") pod \"multus-additional-cni-plugins-q79jt\" (UID: \"6fbaa452-6ab5-4370-ae18-fb4290ce9c10\") " pod="openshift-multus/multus-additional-cni-plugins-q79jt"
Jul 23 14:52:40.992310 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.992189    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e0c66f26-06fe-4273-ba29-fa20ce89b873-host-kubelet\") pod \"ovnkube-node-z4fvn\" (UID: \"e0c66f26-06fe-4273-ba29-fa20ce89b873\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4fvn"
Jul 23 14:52:40.992310 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.992209    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e0c66f26-06fe-4273-ba29-fa20ce89b873-systemd-units\") pod \"ovnkube-node-z4fvn\" (UID: \"e0c66f26-06fe-4273-ba29-fa20ce89b873\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4fvn"
Jul 23 14:52:40.992310 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.992247    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/37ed7912-d183-41e3-994b-561a8815f7d5-etc-modprobe-d\") pod \"tuned-rmbz7\" (UID: \"37ed7912-d183-41e3-994b-561a8815f7d5\") " pod="openshift-cluster-node-tuning-operator/tuned-rmbz7"
Jul 23 14:52:40.992310 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.992275    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6fbaa452-6ab5-4370-ae18-fb4290ce9c10-system-cni-dir\") pod \"multus-additional-cni-plugins-q79jt\" (UID: \"6fbaa452-6ab5-4370-ae18-fb4290ce9c10\") " pod="openshift-multus/multus-additional-cni-plugins-q79jt"
Jul 23 14:52:40.992310 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.992300    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e0c66f26-06fe-4273-ba29-fa20ce89b873-log-socket\") pod \"ovnkube-node-z4fvn\" (UID: \"e0c66f26-06fe-4273-ba29-fa20ce89b873\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4fvn"
Jul 23 14:52:40.992310 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.992315    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d9b3550d-e82f-4934-b093-486563f240fb-hosts-file\") pod \"node-resolver-vfsh6\" (UID: \"d9b3550d-e82f-4934-b093-486563f240fb\") " pod="openshift-dns/node-resolver-vfsh6"
Jul 23 14:52:40.992835 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.992336    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bc81888a-6094-4c2e-b881-6b9cd628e6c1-system-cni-dir\") pod \"multus-68mqt\" (UID: \"bc81888a-6094-4c2e-b881-6b9cd628e6c1\") " pod="openshift-multus/multus-68mqt"
Jul 23 14:52:40.992835 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.992357    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bc81888a-6094-4c2e-b881-6b9cd628e6c1-host-run-netns\") pod \"multus-68mqt\" (UID: \"bc81888a-6094-4c2e-b881-6b9cd628e6c1\") " pod="openshift-multus/multus-68mqt"
Jul 23 14:52:40.992835 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.992378    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bc81888a-6094-4c2e-b881-6b9cd628e6c1-etc-kubernetes\") pod \"multus-68mqt\" (UID: \"bc81888a-6094-4c2e-b881-6b9cd628e6c1\") " pod="openshift-multus/multus-68mqt"
Jul 23 14:52:40.992835 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.992429    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e0c66f26-06fe-4273-ba29-fa20ce89b873-log-socket\") pod \"ovnkube-node-z4fvn\" (UID: \"e0c66f26-06fe-4273-ba29-fa20ce89b873\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4fvn"
Jul 23 14:52:40.992835 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.992419    2560 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 23 14:52:40.992835 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.992497    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/37ed7912-d183-41e3-994b-561a8815f7d5-etc-modprobe-d\") pod \"tuned-rmbz7\" (UID: \"37ed7912-d183-41e3-994b-561a8815f7d5\") " pod="openshift-cluster-node-tuning-operator/tuned-rmbz7"
Jul 23 14:52:40.992835 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.992500    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9a2b87d6-d3dd-41b8-96f8-2e7a0be09325-host\") pod \"node-ca-lz8qd\" (UID: \"9a2b87d6-d3dd-41b8-96f8-2e7a0be09325\") " pod="openshift-image-registry/node-ca-lz8qd"
Jul 23 14:52:40.992835 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.992534    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/37ed7912-d183-41e3-994b-561a8815f7d5-var-lib-kubelet\") pod \"tuned-rmbz7\" (UID: \"37ed7912-d183-41e3-994b-561a8815f7d5\") " pod="openshift-cluster-node-tuning-operator/tuned-rmbz7"
Jul 23 14:52:40.992835 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.992547    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e0c66f26-06fe-4273-ba29-fa20ce89b873-systemd-units\") pod \"ovnkube-node-z4fvn\" (UID: \"e0c66f26-06fe-4273-ba29-fa20ce89b873\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4fvn"
Jul 23 14:52:40.992835 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.992561    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/95f95dc6-4b74-428b-8d51-d7975d5ed0e2-sys-fs\") pod \"aws-ebs-csi-driver-node-9qk7h\" (UID: \"95f95dc6-4b74-428b-8d51-d7975d5ed0e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9qk7h"
Jul 23 14:52:40.992835 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.992586    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e0c66f26-06fe-4273-ba29-fa20ce89b873-var-lib-openvswitch\") pod \"ovnkube-node-z4fvn\" (UID: \"e0c66f26-06fe-4273-ba29-fa20ce89b873\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4fvn"
Jul 23 14:52:40.992835 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.992589    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e0c66f26-06fe-4273-ba29-fa20ce89b873-host-kubelet\") pod \"ovnkube-node-z4fvn\" (UID: \"e0c66f26-06fe-4273-ba29-fa20ce89b873\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4fvn"
Jul 23 14:52:40.992835 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.992638    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e0c66f26-06fe-4273-ba29-fa20ce89b873-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-z4fvn\" (UID: \"e0c66f26-06fe-4273-ba29-fa20ce89b873\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4fvn"
Jul 23 14:52:40.992835 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.992645    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/37ed7912-d183-41e3-994b-561a8815f7d5-var-lib-kubelet\") pod \"tuned-rmbz7\" (UID: \"37ed7912-d183-41e3-994b-561a8815f7d5\") " pod="openshift-cluster-node-tuning-operator/tuned-rmbz7"
Jul 23 14:52:40.992835 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.992671    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/37ed7912-d183-41e3-994b-561a8815f7d5-etc-tuned\") pod \"tuned-rmbz7\" (UID: \"37ed7912-d183-41e3-994b-561a8815f7d5\") " pod="openshift-cluster-node-tuning-operator/tuned-rmbz7"
Jul 23 14:52:40.993438 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.992934    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9a2b87d6-d3dd-41b8-96f8-2e7a0be09325-serviceca\") pod \"node-ca-lz8qd\" (UID: \"9a2b87d6-d3dd-41b8-96f8-2e7a0be09325\") " pod="openshift-image-registry/node-ca-lz8qd"
Jul 23 14:52:40.993438 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.992971    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/95f95dc6-4b74-428b-8d51-d7975d5ed0e2-etc-selinux\") pod \"aws-ebs-csi-driver-node-9qk7h\" (UID: \"95f95dc6-4b74-428b-8d51-d7975d5ed0e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9qk7h"
Jul 23 14:52:40.993438 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.992999    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bc81888a-6094-4c2e-b881-6b9cd628e6c1-multus-socket-dir-parent\") pod \"multus-68mqt\" (UID: \"bc81888a-6094-4c2e-b881-6b9cd628e6c1\") " pod="openshift-multus/multus-68mqt"
Jul 23 14:52:40.993438 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.993032    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bc81888a-6094-4c2e-b881-6b9cd628e6c1-host-var-lib-cni-multus\") pod \"multus-68mqt\" (UID: \"bc81888a-6094-4c2e-b881-6b9cd628e6c1\") " pod="openshift-multus/multus-68mqt"
Jul 23 14:52:40.993438 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.993049    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bc81888a-6094-4c2e-b881-6b9cd628e6c1-host-var-lib-kubelet\") pod \"multus-68mqt\" (UID: \"bc81888a-6094-4c2e-b881-6b9cd628e6c1\") " pod="openshift-multus/multus-68mqt"
Jul 23 14:52:40.993438 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.993077    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e0c66f26-06fe-4273-ba29-fa20ce89b873-run-ovn\") pod \"ovnkube-node-z4fvn\" (UID: \"e0c66f26-06fe-4273-ba29-fa20ce89b873\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4fvn"
Jul 23 14:52:40.993438 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.993111    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/37ed7912-d183-41e3-994b-561a8815f7d5-etc-kubernetes\") pod \"tuned-rmbz7\" (UID: \"37ed7912-d183-41e3-994b-561a8815f7d5\") " pod="openshift-cluster-node-tuning-operator/tuned-rmbz7"
Jul 23 14:52:40.993438 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.993141    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5gs7\" (UniqueName: \"kubernetes.io/projected/37ed7912-d183-41e3-994b-561a8815f7d5-kube-api-access-q5gs7\") pod \"tuned-rmbz7\" (UID: \"37ed7912-d183-41e3-994b-561a8815f7d5\") " pod="openshift-cluster-node-tuning-operator/tuned-rmbz7"
Jul 23 14:52:40.993438 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.993182    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9b3550d-e82f-4934-b093-486563f240fb-tmp-dir\") pod \"node-resolver-vfsh6\" (UID: \"d9b3550d-e82f-4934-b093-486563f240fb\") " pod="openshift-dns/node-resolver-vfsh6"
Jul 23 14:52:40.993438 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.993213    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bc81888a-6094-4c2e-b881-6b9cd628e6c1-host-run-k8s-cni-cncf-io\") pod \"multus-68mqt\" (UID: \"bc81888a-6094-4c2e-b881-6b9cd628e6c1\") " pod="openshift-multus/multus-68mqt"
Jul 23 14:52:40.993770 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.993650    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9a2b87d6-d3dd-41b8-96f8-2e7a0be09325-serviceca\") pod \"node-ca-lz8qd\" (UID: \"9a2b87d6-d3dd-41b8-96f8-2e7a0be09325\") " pod="openshift-image-registry/node-ca-lz8qd"
Jul 23 14:52:40.993770 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.993711    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e0c66f26-06fe-4273-ba29-fa20ce89b873-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-z4fvn\" (UID: \"e0c66f26-06fe-4273-ba29-fa20ce89b873\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4fvn"
Jul 23 14:52:40.993840 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.993766    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e0c66f26-06fe-4273-ba29-fa20ce89b873-run-ovn\") pod \"ovnkube-node-z4fvn\" (UID: \"e0c66f26-06fe-4273-ba29-fa20ce89b873\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4fvn"
Jul 23 14:52:40.993840 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.993832    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/37ed7912-d183-41e3-994b-561a8815f7d5-etc-kubernetes\") pod \"tuned-rmbz7\" (UID: \"37ed7912-d183-41e3-994b-561a8815f7d5\") " pod="openshift-cluster-node-tuning-operator/tuned-rmbz7"
Jul 23 14:52:40.993960 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.993943    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e0c66f26-06fe-4273-ba29-fa20ce89b873-var-lib-openvswitch\") pod \"ovnkube-node-z4fvn\" (UID: \"e0c66f26-06fe-4273-ba29-fa20ce89b873\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4fvn"
Jul 23 14:52:40.994097 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.994056    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bc81888a-6094-4c2e-b881-6b9cd628e6c1-multus-conf-dir\") pod \"multus-68mqt\" (UID: \"bc81888a-6094-4c2e-b881-6b9cd628e6c1\") " pod="openshift-multus/multus-68mqt"
Jul 23 14:52:40.994168 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.994103    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6fbaa452-6ab5-4370-ae18-fb4290ce9c10-tuning-conf-dir\") pod \"multus-additional-cni-plugins-q79jt\" (UID: \"6fbaa452-6ab5-4370-ae18-fb4290ce9c10\") " pod="openshift-multus/multus-additional-cni-plugins-q79jt"
Jul 23 14:52:40.994168 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.994142    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/95f95dc6-4b74-428b-8d51-d7975d5ed0e2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9qk7h\" (UID: \"95f95dc6-4b74-428b-8d51-d7975d5ed0e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9qk7h"
Jul 23 14:52:40.994288 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.994196    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dbjr\" (UniqueName: \"kubernetes.io/projected/c4471b3d-89b5-46d0-b16d-734e52f04801-kube-api-access-4dbjr\") pod \"iptables-alerter-gd6w9\" (UID: \"c4471b3d-89b5-46d0-b16d-734e52f04801\") " pod="openshift-network-operator/iptables-alerter-gd6w9"
Jul 23 14:52:40.994288 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.994256    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bc81888a-6094-4c2e-b881-6b9cd628e6c1-host-run-multus-certs\") pod \"multus-68mqt\" (UID: \"bc81888a-6094-4c2e-b881-6b9cd628e6c1\") " pod="openshift-multus/multus-68mqt"
Jul 23 14:52:40.994378 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.994306    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b3df911b-3c63-441b-97fa-8d431bf3fb95-agent-certs\") pod \"konnectivity-agent-ns8qd\" (UID: \"b3df911b-3c63-441b-97fa-8d431bf3fb95\") " pod="kube-system/konnectivity-agent-ns8qd"
Jul 23 14:52:40.994427 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.994384    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e0c66f26-06fe-4273-ba29-fa20ce89b873-host-slash\") pod \"ovnkube-node-z4fvn\" (UID: \"e0c66f26-06fe-4273-ba29-fa20ce89b873\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4fvn"
Jul 23 14:52:40.994478 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.994456    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e0c66f26-06fe-4273-ba29-fa20ce89b873-env-overrides\") pod \"ovnkube-node-z4fvn\" (UID: \"e0c66f26-06fe-4273-ba29-fa20ce89b873\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4fvn"
Jul 23 14:52:40.994525 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.994490    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e0c66f26-06fe-4273-ba29-fa20ce89b873-ovn-node-metrics-cert\") pod \"ovnkube-node-z4fvn\" (UID: \"e0c66f26-06fe-4273-ba29-fa20ce89b873\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4fvn"
Jul 23 14:52:40.994835 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.994805    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e0c66f26-06fe-4273-ba29-fa20ce89b873-env-overrides\") pod \"ovnkube-node-z4fvn\" (UID: \"e0c66f26-06fe-4273-ba29-fa20ce89b873\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4fvn"
Jul 23 14:52:40.994990 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.994858    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bc81888a-6094-4c2e-b881-6b9cd628e6c1-cnibin\") pod \"multus-68mqt\" (UID: \"bc81888a-6094-4c2e-b881-6b9cd628e6c1\") " pod="openshift-multus/multus-68mqt"
Jul 23 14:52:40.994990 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.994890    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/bc81888a-6094-4c2e-b881-6b9cd628e6c1-os-release\") pod \"multus-68mqt\" (UID: \"bc81888a-6094-4c2e-b881-6b9cd628e6c1\") " pod="openshift-multus/multus-68mqt"
Jul 23 14:52:40.994990 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.994920    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/37ed7912-d183-41e3-994b-561a8815f7d5-etc-sysconfig\") pod \"tuned-rmbz7\" (UID: \"37ed7912-d183-41e3-994b-561a8815f7d5\") " pod="openshift-cluster-node-tuning-operator/tuned-rmbz7"
Jul 23 14:52:40.994990 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.994973    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/527775ad-7c13-4732-b824-4302ad79dcf4-dbus\") pod \"global-pull-secret-syncer-hvsnb\" (UID: \"527775ad-7c13-4732-b824-4302ad79dcf4\") " pod="kube-system/global-pull-secret-syncer-hvsnb"
Jul 23 14:52:40.995194 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.995003    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/527775ad-7c13-4732-b824-4302ad79dcf4-original-pull-secret\") pod \"global-pull-secret-syncer-hvsnb\" (UID: \"527775ad-7c13-4732-b824-4302ad79dcf4\") " pod="kube-system/global-pull-secret-syncer-hvsnb"
Jul 23 14:52:40.995194 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.995035    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/37ed7912-d183-41e3-994b-561a8815f7d5-etc-sysctl-d\") pod \"tuned-rmbz7\" (UID: \"37ed7912-d183-41e3-994b-561a8815f7d5\") " pod="openshift-cluster-node-tuning-operator/tuned-rmbz7"
Jul 23 14:52:40.995194 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.995062    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/37ed7912-d183-41e3-994b-561a8815f7d5-etc-sysctl-conf\") pod \"tuned-rmbz7\" (UID: \"37ed7912-d183-41e3-994b-561a8815f7d5\") " pod="openshift-cluster-node-tuning-operator/tuned-rmbz7"
Jul 23 14:52:40.995194 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.995092    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/37ed7912-d183-41e3-994b-561a8815f7d5-host\") pod \"tuned-rmbz7\" (UID: \"37ed7912-d183-41e3-994b-561a8815f7d5\") " pod="openshift-cluster-node-tuning-operator/tuned-rmbz7"
Jul 23 14:52:40.995194 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.995124    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcqwm\" (UniqueName: \"kubernetes.io/projected/3370857a-3ffb-4603-bcad-503b48c10fd6-kube-api-access-fcqwm\") pod \"network-check-target-7sqtc\" (UID: \"3370857a-3ffb-4603-bcad-503b48c10fd6\") " pod="openshift-network-diagnostics/network-check-target-7sqtc"
Jul 23 14:52:40.995453 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.995216    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/37ed7912-d183-41e3-994b-561a8815f7d5-host\") pod \"tuned-rmbz7\" (UID: \"37ed7912-d183-41e3-994b-561a8815f7d5\") " pod="openshift-cluster-node-tuning-operator/tuned-rmbz7"
Jul 23 14:52:40.995453 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.995275    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bc81888a-6094-4c2e-b881-6b9cd628e6c1-hostroot\") pod \"multus-68mqt\" (UID: \"bc81888a-6094-4c2e-b881-6b9cd628e6c1\") " pod="openshift-multus/multus-68mqt"
Jul 23 14:52:40.995453 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.995306    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/413d2464-7258-4246-85bb-ac9f03976ee5-metrics-certs\") pod \"network-metrics-daemon-hvw8v\" (UID: \"413d2464-7258-4246-85bb-ac9f03976ee5\") " pod="openshift-multus/network-metrics-daemon-hvw8v"
Jul 23 14:52:40.995453 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.995338    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e0c66f26-06fe-4273-ba29-fa20ce89b873-node-log\") pod \"ovnkube-node-z4fvn\" (UID: \"e0c66f26-06fe-4273-ba29-fa20ce89b873\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4fvn"
Jul 23 14:52:40.995453 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.995369    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e0c66f26-06fe-4273-ba29-fa20ce89b873-host-cni-bin\") pod \"ovnkube-node-z4fvn\" (UID: \"e0c66f26-06fe-4273-ba29-fa20ce89b873\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4fvn"
Jul 23 14:52:40.995705 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.995508    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e0c66f26-06fe-4273-ba29-fa20ce89b873-host-cni-bin\") pod \"ovnkube-node-z4fvn\" (UID: \"e0c66f26-06fe-4273-ba29-fa20ce89b873\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4fvn"
Jul 23 14:52:40.995804 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.995786    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/37ed7912-d183-41e3-994b-561a8815f7d5-etc-sysconfig\") pod \"tuned-rmbz7\" (UID: \"37ed7912-d183-41e3-994b-561a8815f7d5\") " pod="openshift-cluster-node-tuning-operator/tuned-rmbz7"
Jul 23 14:52:40.997147 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.997122    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e0c66f26-06fe-4273-ba29-fa20ce89b873-host-slash\") pod \"ovnkube-node-z4fvn\" (UID: \"e0c66f26-06fe-4273-ba29-fa20ce89b873\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4fvn"
Jul 23 14:52:40.997696 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.997168    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/37ed7912-d183-41e3-994b-561a8815f7d5-etc-sysctl-d\") pod \"tuned-rmbz7\" (UID: \"37ed7912-d183-41e3-994b-561a8815f7d5\") " pod="openshift-cluster-node-tuning-operator/tuned-rmbz7"
Jul 23 14:52:40.997696 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.997218    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e0c66f26-06fe-4273-ba29-fa20ce89b873-node-log\") pod \"ovnkube-node-z4fvn\" (UID: \"e0c66f26-06fe-4273-ba29-fa20ce89b873\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4fvn"
Jul 23 14:52:40.997696 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.997274    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/37ed7912-d183-41e3-994b-561a8815f7d5-etc-sysctl-conf\") pod \"tuned-rmbz7\" (UID: \"37ed7912-d183-41e3-994b-561a8815f7d5\") " pod="openshift-cluster-node-tuning-operator/tuned-rmbz7"
Jul 23 14:52:40.997696 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:52:40.997287    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 14:52:40.997696 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:52:40.997435    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/413d2464-7258-4246-85bb-ac9f03976ee5-metrics-certs podName:413d2464-7258-4246-85bb-ac9f03976ee5 nodeName:}" failed. No retries permitted until 2026-07-23 14:52:41.497389053 +0000 UTC m=+3.005262793 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/413d2464-7258-4246-85bb-ac9f03976ee5-metrics-certs") pod "network-metrics-daemon-hvw8v" (UID: "413d2464-7258-4246-85bb-ac9f03976ee5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 14:52:40.997696 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.997561    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/37ed7912-d183-41e3-994b-561a8815f7d5-etc-tuned\") pod \"tuned-rmbz7\" (UID: \"37ed7912-d183-41e3-994b-561a8815f7d5\") " pod="openshift-cluster-node-tuning-operator/tuned-rmbz7"
Jul 23 14:52:40.998503 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.998480    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e0c66f26-06fe-4273-ba29-fa20ce89b873-ovn-node-metrics-cert\") pod \"ovnkube-node-z4fvn\" (UID: \"e0c66f26-06fe-4273-ba29-fa20ce89b873\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4fvn"
Jul 23 14:52:40.998899 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:40.998880    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/37ed7912-d183-41e3-994b-561a8815f7d5-tmp\") pod \"tuned-rmbz7\" (UID: \"37ed7912-d183-41e3-994b-561a8815f7d5\") " pod="openshift-cluster-node-tuning-operator/tuned-rmbz7"
Jul 23 14:52:41.002908 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:52:41.002769    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 23 14:52:41.002908 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:52:41.002792    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 23 14:52:41.002908 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:52:41.002805    2560 projected.go:196] Error preparing data for projected volume kube-api-access-fcqwm for pod openshift-network-diagnostics/network-check-target-7sqtc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 14:52:41.002908 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:52:41.002871    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3370857a-3ffb-4603-bcad-503b48c10fd6-kube-api-access-fcqwm podName:3370857a-3ffb-4603-bcad-503b48c10fd6 nodeName:}" failed. No retries permitted until 2026-07-23 14:52:41.502854315 +0000 UTC m=+3.010728044 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-fcqwm" (UniqueName: "kubernetes.io/projected/3370857a-3ffb-4603-bcad-503b48c10fd6-kube-api-access-fcqwm") pod "network-check-target-7sqtc" (UID: "3370857a-3ffb-4603-bcad-503b48c10fd6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 14:52:41.005066 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:41.005046    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r266g\" (UniqueName: \"kubernetes.io/projected/e0c66f26-06fe-4273-ba29-fa20ce89b873-kube-api-access-r266g\") pod \"ovnkube-node-z4fvn\" (UID: \"e0c66f26-06fe-4273-ba29-fa20ce89b873\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4fvn"
Jul 23 14:52:41.005521 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:41.005434    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5gs7\" (UniqueName: \"kubernetes.io/projected/37ed7912-d183-41e3-994b-561a8815f7d5-kube-api-access-q5gs7\") pod \"tuned-rmbz7\" (UID: \"37ed7912-d183-41e3-994b-561a8815f7d5\") " pod="openshift-cluster-node-tuning-operator/tuned-rmbz7"
Jul 23 14:52:41.006106 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:41.006078    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2hgq\" (UniqueName: \"kubernetes.io/projected/9a2b87d6-d3dd-41b8-96f8-2e7a0be09325-kube-api-access-r2hgq\") pod \"node-ca-lz8qd\" (UID: \"9a2b87d6-d3dd-41b8-96f8-2e7a0be09325\") " pod="openshift-image-registry/node-ca-lz8qd"
Jul 23 14:52:41.006189 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:41.006110    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxfps\" (UniqueName: \"kubernetes.io/projected/413d2464-7258-4246-85bb-ac9f03976ee5-kube-api-access-gxfps\") pod \"network-metrics-daemon-hvw8v\" (UID: \"413d2464-7258-4246-85bb-ac9f03976ee5\") " pod="openshift-multus/network-metrics-daemon-hvw8v"
Jul 23 14:52:41.096271 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:41.096218    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcldm\" (UniqueName: \"kubernetes.io/projected/95f95dc6-4b74-428b-8d51-d7975d5ed0e2-kube-api-access-xcldm\") pod \"aws-ebs-csi-driver-node-9qk7h\" (UID: \"95f95dc6-4b74-428b-8d51-d7975d5ed0e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9qk7h"
Jul 23 14:52:41.096434 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:41.096278    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/527775ad-7c13-4732-b824-4302ad79dcf4-kubelet-config\") pod \"global-pull-secret-syncer-hvsnb\" (UID: \"527775ad-7c13-4732-b824-4302ad79dcf4\") " pod="kube-system/global-pull-secret-syncer-hvsnb"
Jul 23 14:52:41.096434 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:41.096306    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c4471b3d-89b5-46d0-b16d-734e52f04801-host-slash\") pod \"iptables-alerter-gd6w9\" (UID: \"c4471b3d-89b5-46d0-b16d-734e52f04801\") " pod="openshift-network-operator/iptables-alerter-gd6w9"
Jul 23 14:52:41.096434 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:41.096351    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bc81888a-6094-4c2e-b881-6b9cd628e6c1-cni-binary-copy\") pod \"multus-68mqt\" (UID: \"bc81888a-6094-4c2e-b881-6b9cd628e6c1\") " pod="openshift-multus/multus-68mqt"
Jul 23 14:52:41.096434 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:41.096374    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bc81888a-6094-4c2e-b881-6b9cd628e6c1-multus-daemon-config\") pod \"multus-68mqt\" (UID: \"bc81888a-6094-4c2e-b881-6b9cd628e6c1\") " pod="openshift-multus/multus-68mqt"
Jul 23 14:52:41.096434 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:41.096397    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8d4t\" (UniqueName: \"kubernetes.io/projected/bc81888a-6094-4c2e-b881-6b9cd628e6c1-kube-api-access-x8d4t\") pod \"multus-68mqt\" (UID: \"bc81888a-6094-4c2e-b881-6b9cd628e6c1\") " pod="openshift-multus/multus-68mqt"
Jul 23 14:52:41.096434 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:41.096420    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6fbaa452-6ab5-4370-ae18-fb4290ce9c10-cni-binary-copy\") pod \"multus-additional-cni-plugins-q79jt\" (UID: \"6fbaa452-6ab5-4370-ae18-fb4290ce9c10\") " pod="openshift-multus/multus-additional-cni-plugins-q79jt"
Jul 23 14:52:41.096434 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:41.096422    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/527775ad-7c13-4732-b824-4302ad79dcf4-kubelet-config\") pod \"global-pull-secret-syncer-hvsnb\" (UID: \"527775ad-7c13-4732-b824-4302ad79dcf4\") " pod="kube-system/global-pull-secret-syncer-hvsnb"
Jul 23 14:52:41.096770 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:41.096446    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6nwq\" (UniqueName: \"kubernetes.io/projected/d9b3550d-e82f-4934-b093-486563f240fb-kube-api-access-d6nwq\") pod \"node-resolver-vfsh6\" (UID: \"d9b3550d-e82f-4934-b093-486563f240fb\") " pod="openshift-dns/node-resolver-vfsh6"
Jul 23 14:52:41.096770 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:41.096470    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bc81888a-6094-4c2e-b881-6b9cd628e6c1-host-var-lib-cni-bin\") pod \"multus-68mqt\" (UID: \"bc81888a-6094-4c2e-b881-6b9cd628e6c1\") " pod="openshift-multus/multus-68mqt"
Jul 23 14:52:41.096770 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:41.096497    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6fbaa452-6ab5-4370-ae18-fb4290ce9c10-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-q79jt\" (UID: \"6fbaa452-6ab5-4370-ae18-fb4290ce9c10\") " pod="openshift-multus/multus-additional-cni-plugins-q79jt"
Jul 23 14:52:41.096770 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:41.096523    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/95f95dc6-4b74-428b-8d51-d7975d5ed0e2-socket-dir\") pod \"aws-ebs-csi-driver-node-9qk7h\" (UID: \"95f95dc6-4b74-428b-8d51-d7975d5ed0e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9qk7h"
Jul 23 14:52:41.096770 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:41.096545    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6fbaa452-6ab5-4370-ae18-fb4290ce9c10-cnibin\") pod \"multus-additional-cni-plugins-q79jt\" (UID: \"6fbaa452-6ab5-4370-ae18-fb4290ce9c10\") " pod="openshift-multus/multus-additional-cni-plugins-q79jt"
Jul 23 14:52:41.096770 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:41.096568    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6fbaa452-6ab5-4370-ae18-fb4290ce9c10-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-q79jt\" (UID: \"6fbaa452-6ab5-4370-ae18-fb4290ce9c10\") " pod="openshift-multus/multus-additional-cni-plugins-q79jt"
Jul 23 14:52:41.096770 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:41.096593    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ht5cd\" (UniqueName: \"kubernetes.io/projected/6fbaa452-6ab5-4370-ae18-fb4290ce9c10-kube-api-access-ht5cd\") pod \"multus-additional-cni-plugins-q79jt\" (UID: \"6fbaa452-6ab5-4370-ae18-fb4290ce9c10\") " pod="openshift-multus/multus-additional-cni-plugins-q79jt"
Jul 23 14:52:41.096770 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:41.096619    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bc81888a-6094-4c2e-b881-6b9cd628e6c1-multus-cni-dir\") pod \"multus-68mqt\" (UID: \"bc81888a-6094-4c2e-b881-6b9cd628e6c1\") " pod="openshift-multus/multus-68mqt"
Jul 23 14:52:41.096770 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:41.096642    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b3df911b-3c63-441b-97fa-8d431bf3fb95-konnectivity-ca\") pod \"konnectivity-agent-ns8qd\" (UID: \"b3df911b-3c63-441b-97fa-8d431bf3fb95\") " pod="kube-system/konnectivity-agent-ns8qd"
Jul 23 14:52:41.096770 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:41.096689    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/95f95dc6-4b74-428b-8d51-d7975d5ed0e2-registration-dir\") pod \"aws-ebs-csi-driver-node-9qk7h\" (UID: \"95f95dc6-4b74-428b-8d51-d7975d5ed0e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9qk7h"
Jul 23 14:52:41.096770 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:41.096714    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c4471b3d-89b5-46d0-b16d-734e52f04801-iptables-alerter-script\") pod \"iptables-alerter-gd6w9\" (UID: \"c4471b3d-89b5-46d0-b16d-734e52f04801\") " pod="openshift-network-operator/iptables-alerter-gd6w9"
Jul 23 14:52:41.096770 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:41.096742    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/95f95dc6-4b74-428b-8d51-d7975d5ed0e2-device-dir\") pod \"aws-ebs-csi-driver-node-9qk7h\" (UID: \"95f95dc6-4b74-428b-8d51-d7975d5ed0e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9qk7h"
Jul 23 14:52:41.096770 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:41.096764    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6fbaa452-6ab5-4370-ae18-fb4290ce9c10-os-release\") pod \"multus-additional-cni-plugins-q79jt\" (UID: \"6fbaa452-6ab5-4370-ae18-fb4290ce9c10\") " pod="openshift-multus/multus-additional-cni-plugins-q79jt"
Jul 23 14:52:41.097397 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:41.096791    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6fbaa452-6ab5-4370-ae18-fb4290ce9c10-system-cni-dir\") pod \"multus-additional-cni-plugins-q79jt\" (UID: \"6fbaa452-6ab5-4370-ae18-fb4290ce9c10\") " pod="openshift-multus/multus-additional-cni-plugins-q79jt"
Jul 23 14:52:41.097397 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:41.096817    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d9b3550d-e82f-4934-b093-486563f240fb-hosts-file\") pod \"node-resolver-vfsh6\" (UID: \"d9b3550d-e82f-4934-b093-486563f240fb\") " pod="openshift-dns/node-resolver-vfsh6"
Jul 23 14:52:41.097397 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:41.096841    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bc81888a-6094-4c2e-b881-6b9cd628e6c1-system-cni-dir\") pod \"multus-68mqt\" (UID: \"bc81888a-6094-4c2e-b881-6b9cd628e6c1\") " pod="openshift-multus/multus-68mqt"
Jul 23 14:52:41.097397 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:41.096869    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bc81888a-6094-4c2e-b881-6b9cd628e6c1-host-run-netns\") pod \"multus-68mqt\" (UID: \"bc81888a-6094-4c2e-b881-6b9cd628e6c1\") " pod="openshift-multus/multus-68mqt"
Jul 23 14:52:41.097397 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:41.096893    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bc81888a-6094-4c2e-b881-6b9cd628e6c1-etc-kubernetes\") pod \"multus-68mqt\" (UID: \"bc81888a-6094-4c2e-b881-6b9cd628e6c1\") " pod="openshift-multus/multus-68mqt"
Jul 23 14:52:41.097397 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:41.096932    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/95f95dc6-4b74-428b-8d51-d7975d5ed0e2-sys-fs\") pod \"aws-ebs-csi-driver-node-9qk7h\" (UID: \"95f95dc6-4b74-428b-8d51-d7975d5ed0e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9qk7h"
Jul 23 14:52:41.097397 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:41.096972    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/95f95dc6-4b74-428b-8d51-d7975d5ed0e2-etc-selinux\") pod \"aws-ebs-csi-driver-node-9qk7h\" (UID: \"95f95dc6-4b74-428b-8d51-d7975d5ed0e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9qk7h"
Jul 23 14:52:41.097397 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:41.097010    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bc81888a-6094-4c2e-b881-6b9cd628e6c1-multus-socket-dir-parent\") pod \"multus-68mqt\" (UID: \"bc81888a-6094-4c2e-b881-6b9cd628e6c1\") " pod="openshift-multus/multus-68mqt"
Jul 23 14:52:41.097397 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:41.097021    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bc81888a-6094-4c2e-b881-6b9cd628e6c1-multus-daemon-config\") pod \"multus-68mqt\" (UID: \"bc81888a-6094-4c2e-b881-6b9cd628e6c1\") " pod="openshift-multus/multus-68mqt"
Jul 23 14:52:41.097397 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:41.097033    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bc81888a-6094-4c2e-b881-6b9cd628e6c1-host-var-lib-cni-multus\") pod \"multus-68mqt\" (UID: \"bc81888a-6094-4c2e-b881-6b9cd628e6c1\") " pod="openshift-multus/multus-68mqt"
Jul 23 14:52:41.097397 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:41.097054    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6fbaa452-6ab5-4370-ae18-fb4290ce9c10-cni-binary-copy\") pod \"multus-additional-cni-plugins-q79jt\" (UID: \"6fbaa452-6ab5-4370-ae18-fb4290ce9c10\") " pod="openshift-multus/multus-additional-cni-plugins-q79jt"
Jul 23 14:52:41.097397 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:41.097058    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bc81888a-6094-4c2e-b881-6b9cd628e6c1-host-var-lib-kubelet\") pod \"multus-68mqt\" (UID: \"bc81888a-6094-4c2e-b881-6b9cd628e6c1\") " pod="openshift-multus/multus-68mqt"
Jul 23 14:52:41.097397 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:41.097085    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c4471b3d-89b5-46d0-b16d-734e52f04801-host-slash\") pod \"iptables-alerter-gd6w9\" (UID: \"c4471b3d-89b5-46d0-b16d-734e52f04801\") " pod="openshift-network-operator/iptables-alerter-gd6w9"
Jul 23 14:52:41.097397 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:41.097104    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9b3550d-e82f-4934-b093-486563f240fb-tmp-dir\") pod \"node-resolver-vfsh6\" (UID: \"d9b3550d-e82f-4934-b093-486563f240fb\") " pod="openshift-dns/node-resolver-vfsh6"
Jul 23 14:52:41.097397 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:41.097133    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bc81888a-6094-4c2e-b881-6b9cd628e6c1-host-run-k8s-cni-cncf-io\") pod \"multus-68mqt\" (UID: \"bc81888a-6094-4c2e-b881-6b9cd628e6c1\") " pod="openshift-multus/multus-68mqt"
Jul 23 14:52:41.097397 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:41.097151    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bc81888a-6094-4c2e-b881-6b9cd628e6c1-host-var-lib-cni-bin\") pod \"multus-68mqt\" (UID: \"bc81888a-6094-4c2e-b881-6b9cd628e6c1\") " pod="openshift-multus/multus-68mqt"
Jul 23 14:52:41.097397 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:41.097161    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bc81888a-6094-4c2e-b881-6b9cd628e6c1-multus-conf-dir\") pod \"multus-68mqt\" (UID: \"bc81888a-6094-4c2e-b881-6b9cd628e6c1\") " pod="openshift-multus/multus-68mqt"
Jul 23 14:52:41.098169 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:41.097187    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6fbaa452-6ab5-4370-ae18-fb4290ce9c10-tuning-conf-dir\") pod \"multus-additional-cni-plugins-q79jt\" (UID: \"6fbaa452-6ab5-4370-ae18-fb4290ce9c10\") " pod="openshift-multus/multus-additional-cni-plugins-q79jt"
Jul 23 14:52:41.098169 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:41.097213    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/95f95dc6-4b74-428b-8d51-d7975d5ed0e2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9qk7h\" (UID: \"95f95dc6-4b74-428b-8d51-d7975d5ed0e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9qk7h"
Jul 23 14:52:41.098169 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:41.097264    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dbjr\" (UniqueName: \"kubernetes.io/projected/c4471b3d-89b5-46d0-b16d-734e52f04801-kube-api-access-4dbjr\") pod \"iptables-alerter-gd6w9\" (UID: \"c4471b3d-89b5-46d0-b16d-734e52f04801\") " pod="openshift-network-operator/iptables-alerter-gd6w9"
Jul 23 14:52:41.098169 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:41.097289    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bc81888a-6094-4c2e-b881-6b9cd628e6c1-host-run-multus-certs\") pod \"multus-68mqt\" (UID: \"bc81888a-6094-4c2e-b881-6b9cd628e6c1\") " pod="openshift-multus/multus-68mqt"
Jul 23 14:52:41.098169 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:41.097319    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b3df911b-3c63-441b-97fa-8d431bf3fb95-agent-certs\") pod \"konnectivity-agent-ns8qd\" (UID: \"b3df911b-3c63-441b-97fa-8d431bf3fb95\") " pod="kube-system/konnectivity-agent-ns8qd"
Jul 23 14:52:41.098169 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:41.097346    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bc81888a-6094-4c2e-b881-6b9cd628e6c1-cnibin\") pod \"multus-68mqt\" (UID: \"bc81888a-6094-4c2e-b881-6b9cd628e6c1\") " pod="openshift-multus/multus-68mqt"
Jul 23 14:52:41.098169 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:41.097370    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/bc81888a-6094-4c2e-b881-6b9cd628e6c1-os-release\") pod \"multus-68mqt\" (UID: \"bc81888a-6094-4c2e-b881-6b9cd628e6c1\") " pod="openshift-multus/multus-68mqt"
Jul 23 14:52:41.098169 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:41.097396    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/527775ad-7c13-4732-b824-4302ad79dcf4-dbus\") pod \"global-pull-secret-syncer-hvsnb\" (UID: \"527775ad-7c13-4732-b824-4302ad79dcf4\") " pod="kube-system/global-pull-secret-syncer-hvsnb"
Jul 23 14:52:41.098169 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:41.097425    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/527775ad-7c13-4732-b824-4302ad79dcf4-original-pull-secret\") pod \"global-pull-secret-syncer-hvsnb\" (UID: \"527775ad-7c13-4732-b824-4302ad79dcf4\") " pod="kube-system/global-pull-secret-syncer-hvsnb"
Jul 23 14:52:41.098169 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:41.097464    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bc81888a-6094-4c2e-b881-6b9cd628e6c1-hostroot\") pod \"multus-68mqt\" (UID: \"bc81888a-6094-4c2e-b881-6b9cd628e6c1\") " pod="openshift-multus/multus-68mqt"
Jul 23 14:52:41.098169 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:41.097538    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bc81888a-6094-4c2e-b881-6b9cd628e6c1-cni-binary-copy\") pod \"multus-68mqt\" (UID: \"bc81888a-6094-4c2e-b881-6b9cd628e6c1\") " pod="openshift-multus/multus-68mqt"
Jul 23 14:52:41.098169 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:41.097562    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bc81888a-6094-4c2e-b881-6b9cd628e6c1-hostroot\") pod \"multus-68mqt\" (UID: \"bc81888a-6094-4c2e-b881-6b9cd628e6c1\") " pod="openshift-multus/multus-68mqt"
Jul 23 14:52:41.098169 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:41.097615    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6fbaa452-6ab5-4370-ae18-fb4290ce9c10-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-q79jt\" (UID: \"6fbaa452-6ab5-4370-ae18-fb4290ce9c10\") " pod="openshift-multus/multus-additional-cni-plugins-q79jt"
Jul 23 14:52:41.098169 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:41.097717    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/95f95dc6-4b74-428b-8d51-d7975d5ed0e2-socket-dir\") pod \"aws-ebs-csi-driver-node-9qk7h\" (UID: \"95f95dc6-4b74-428b-8d51-d7975d5ed0e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9qk7h"
Jul 23 14:52:41.098169 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:41.097767    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6fbaa452-6ab5-4370-ae18-fb4290ce9c10-cnibin\") pod \"multus-additional-cni-plugins-q79jt\" (UID: \"6fbaa452-6ab5-4370-ae18-fb4290ce9c10\") " pod="openshift-multus/multus-additional-cni-plugins-q79jt"
Jul 23 14:52:41.098169 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:41.097936    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9b3550d-e82f-4934-b093-486563f240fb-tmp-dir\") pod \"node-resolver-vfsh6\" (UID: \"d9b3550d-e82f-4934-b093-486563f240fb\") " pod="openshift-dns/node-resolver-vfsh6"
Jul 23 14:52:41.098169 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:41.097105    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bc81888a-6094-4c2e-b881-6b9cd628e6c1-host-var-lib-kubelet\") pod \"multus-68mqt\" (UID: \"bc81888a-6094-4c2e-b881-6b9cd628e6c1\") " pod="openshift-multus/multus-68mqt"
Jul 23 14:52:41.098169 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:41.097980    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bc81888a-6094-4c2e-b881-6b9cd628e6c1-host-run-multus-certs\") pod \"multus-68mqt\" (UID: \"bc81888a-6094-4c2e-b881-6b9cd628e6c1\") " pod="openshift-multus/multus-68mqt"
Jul 23 14:52:41.099007 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:41.098025    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bc81888a-6094-4c2e-b881-6b9cd628e6c1-host-run-k8s-cni-cncf-io\") pod \"multus-68mqt\" (UID: \"bc81888a-6094-4c2e-b881-6b9cd628e6c1\") " pod="openshift-multus/multus-68mqt"
Jul 23 14:52:41.099007 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:41.098074    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bc81888a-6094-4c2e-b881-6b9cd628e6c1-multus-conf-dir\") pod \"multus-68mqt\" (UID: \"bc81888a-6094-4c2e-b881-6b9cd628e6c1\") " pod="openshift-multus/multus-68mqt"
Jul 23 14:52:41.099007 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:41.098198    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6fbaa452-6ab5-4370-ae18-fb4290ce9c10-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-q79jt\" (UID: \"6fbaa452-6ab5-4370-ae18-fb4290ce9c10\") " pod="openshift-multus/multus-additional-cni-plugins-q79jt"
Jul 23 14:52:41.099007 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:41.098222    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6fbaa452-6ab5-4370-ae18-fb4290ce9c10-tuning-conf-dir\") pod \"multus-additional-cni-plugins-q79jt\" (UID: \"6fbaa452-6ab5-4370-ae18-fb4290ce9c10\") " pod="openshift-multus/multus-additional-cni-plugins-q79jt"
Jul 23 14:52:41.099007 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:41.098299    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/95f95dc6-4b74-428b-8d51-d7975d5ed0e2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9qk7h\" (UID: \"95f95dc6-4b74-428b-8d51-d7975d5ed0e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9qk7h"
Jul 23 14:52:41.099007 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:41.098312    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/bc81888a-6094-4c2e-b881-6b9cd628e6c1-os-release\") pod \"multus-68mqt\" (UID: \"bc81888a-6094-4c2e-b881-6b9cd628e6c1\") " pod="openshift-multus/multus-68mqt"
Jul 23 14:52:41.099007 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:41.098354    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d9b3550d-e82f-4934-b093-486563f240fb-hosts-file\") pod \"node-resolver-vfsh6\" (UID: \"d9b3550d-e82f-4934-b093-486563f240fb\") " pod="openshift-dns/node-resolver-vfsh6"
Jul 23 14:52:41.099007 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:41.098399    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bc81888a-6094-4c2e-b881-6b9cd628e6c1-cnibin\") pod \"multus-68mqt\" (UID: \"bc81888a-6094-4c2e-b881-6b9cd628e6c1\") " pod="openshift-multus/multus-68mqt"
Jul 23 14:52:41.099007 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:41.098483    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/95f95dc6-4b74-428b-8d51-d7975d5ed0e2-sys-fs\") pod \"aws-ebs-csi-driver-node-9qk7h\" (UID: \"95f95dc6-4b74-428b-8d51-d7975d5ed0e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9qk7h"
Jul 23 14:52:41.099007 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:41.098485    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bc81888a-6094-4c2e-b881-6b9cd628e6c1-multus-cni-dir\") pod \"multus-68mqt\" (UID: \"bc81888a-6094-4c2e-b881-6b9cd628e6c1\") " pod="openshift-multus/multus-68mqt"
Jul 23 14:52:41.099007 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:41.098540    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bc81888a-6094-4c2e-b881-6b9cd628e6c1-system-cni-dir\") pod \"multus-68mqt\" (UID: \"bc81888a-6094-4c2e-b881-6b9cd628e6c1\") " pod="openshift-multus/multus-68mqt"
Jul 23 14:52:41.099007 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:41.098542    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/95f95dc6-4b74-428b-8d51-d7975d5ed0e2-etc-selinux\") pod \"aws-ebs-csi-driver-node-9qk7h\" (UID: \"95f95dc6-4b74-428b-8d51-d7975d5ed0e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9qk7h"
Jul 23 14:52:41.099007 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:41.098589    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bc81888a-6094-4c2e-b881-6b9cd628e6c1-multus-socket-dir-parent\") pod \"multus-68mqt\" (UID: \"bc81888a-6094-4c2e-b881-6b9cd628e6c1\") " pod="openshift-multus/multus-68mqt"
Jul 23 14:52:41.099007 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:41.098633    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bc81888a-6094-4c2e-b881-6b9cd628e6c1-etc-kubernetes\") pod \"multus-68mqt\" (UID: \"bc81888a-6094-4c2e-b881-6b9cd628e6c1\") " pod="openshift-multus/multus-68mqt"
Jul 23 14:52:41.099007 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:41.098678    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bc81888a-6094-4c2e-b881-6b9cd628e6c1-host-var-lib-cni-multus\") pod \"multus-68mqt\" (UID: \"bc81888a-6094-4c2e-b881-6b9cd628e6c1\") " pod="openshift-multus/multus-68mqt"
Jul 23 14:52:41.099007 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:41.098592    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bc81888a-6094-4c2e-b881-6b9cd628e6c1-host-run-netns\") pod \"multus-68mqt\" (UID: \"bc81888a-6094-4c2e-b881-6b9cd628e6c1\") " pod="openshift-multus/multus-68mqt"
Jul 23 14:52:41.099007 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:41.098770    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/95f95dc6-4b74-428b-8d51-d7975d5ed0e2-device-dir\") pod \"aws-ebs-csi-driver-node-9qk7h\" (UID: \"95f95dc6-4b74-428b-8d51-d7975d5ed0e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9qk7h"
Jul 23 14:52:41.099007 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:41.098817    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/95f95dc6-4b74-428b-8d51-d7975d5ed0e2-registration-dir\") pod \"aws-ebs-csi-driver-node-9qk7h\" (UID: \"95f95dc6-4b74-428b-8d51-d7975d5ed0e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9qk7h"
Jul 23 14:52:41.099757 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:41.098874    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/527775ad-7c13-4732-b824-4302ad79dcf4-dbus\") pod \"global-pull-secret-syncer-hvsnb\" (UID: \"527775ad-7c13-4732-b824-4302ad79dcf4\") " pod="kube-system/global-pull-secret-syncer-hvsnb"
Jul 23 14:52:41.099757 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:52:41.098958    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 23 14:52:41.099757 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:41.098996    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b3df911b-3c63-441b-97fa-8d431bf3fb95-konnectivity-ca\") pod \"konnectivity-agent-ns8qd\" (UID: \"b3df911b-3c63-441b-97fa-8d431bf3fb95\") " pod="kube-system/konnectivity-agent-ns8qd"
Jul 23 14:52:41.099757 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:52:41.099022    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/527775ad-7c13-4732-b824-4302ad79dcf4-original-pull-secret podName:527775ad-7c13-4732-b824-4302ad79dcf4 nodeName:}" failed. No retries permitted until 2026-07-23 14:52:41.59900496 +0000 UTC m=+3.106878686 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/527775ad-7c13-4732-b824-4302ad79dcf4-original-pull-secret") pod "global-pull-secret-syncer-hvsnb" (UID: "527775ad-7c13-4732-b824-4302ad79dcf4") : object "kube-system"/"original-pull-secret" not registered
Jul 23 14:52:41.099757 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:41.099070    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6fbaa452-6ab5-4370-ae18-fb4290ce9c10-os-release\") pod \"multus-additional-cni-plugins-q79jt\" (UID: \"6fbaa452-6ab5-4370-ae18-fb4290ce9c10\") " pod="openshift-multus/multus-additional-cni-plugins-q79jt"
Jul 23 14:52:41.099757 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:41.099117    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6fbaa452-6ab5-4370-ae18-fb4290ce9c10-system-cni-dir\") pod \"multus-additional-cni-plugins-q79jt\" (UID: \"6fbaa452-6ab5-4370-ae18-fb4290ce9c10\") " pod="openshift-multus/multus-additional-cni-plugins-q79jt"
Jul 23 14:52:41.099757 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:41.099306    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c4471b3d-89b5-46d0-b16d-734e52f04801-iptables-alerter-script\") pod \"iptables-alerter-gd6w9\" (UID: \"c4471b3d-89b5-46d0-b16d-734e52f04801\") " pod="openshift-network-operator/iptables-alerter-gd6w9"
Jul 23 14:52:41.101044 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:41.101020    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b3df911b-3c63-441b-97fa-8d431bf3fb95-agent-certs\") pod \"konnectivity-agent-ns8qd\" (UID: \"b3df911b-3c63-441b-97fa-8d431bf3fb95\") " pod="kube-system/konnectivity-agent-ns8qd"
Jul 23 14:52:41.107827 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:41.107802    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6nwq\" (UniqueName: \"kubernetes.io/projected/d9b3550d-e82f-4934-b093-486563f240fb-kube-api-access-d6nwq\") pod \"node-resolver-vfsh6\" (UID: \"d9b3550d-e82f-4934-b093-486563f240fb\") " pod="openshift-dns/node-resolver-vfsh6"
Jul 23 14:52:41.107827 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:41.107829    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcldm\" (UniqueName: \"kubernetes.io/projected/95f95dc6-4b74-428b-8d51-d7975d5ed0e2-kube-api-access-xcldm\") pod \"aws-ebs-csi-driver-node-9qk7h\" (UID: \"95f95dc6-4b74-428b-8d51-d7975d5ed0e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9qk7h"
Jul 23 14:52:41.108210 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:41.108193    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8d4t\" (UniqueName: \"kubernetes.io/projected/bc81888a-6094-4c2e-b881-6b9cd628e6c1-kube-api-access-x8d4t\") pod \"multus-68mqt\" (UID: \"bc81888a-6094-4c2e-b881-6b9cd628e6c1\") " pod="openshift-multus/multus-68mqt"
Jul 23 14:52:41.109679 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:41.109657    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ht5cd\" (UniqueName: \"kubernetes.io/projected/6fbaa452-6ab5-4370-ae18-fb4290ce9c10-kube-api-access-ht5cd\") pod \"multus-additional-cni-plugins-q79jt\" (UID: \"6fbaa452-6ab5-4370-ae18-fb4290ce9c10\") " pod="openshift-multus/multus-additional-cni-plugins-q79jt"
Jul 23 14:52:41.109831 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:41.109814    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dbjr\" (UniqueName: \"kubernetes.io/projected/c4471b3d-89b5-46d0-b16d-734e52f04801-kube-api-access-4dbjr\") pod \"iptables-alerter-gd6w9\" (UID: \"c4471b3d-89b5-46d0-b16d-734e52f04801\") " pod="openshift-network-operator/iptables-alerter-gd6w9"
Jul 23 14:52:41.194289 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:41.194196    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-z4fvn"
Jul 23 14:52:41.201480 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:41.201461    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rmbz7"
Jul 23 14:52:41.210111 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:41.210086    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-lz8qd"
Jul 23 14:52:41.215761 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:41.215742    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-q79jt"
Jul 23 14:52:41.224293 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:41.224276    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-vfsh6"
Jul 23 14:52:41.230955 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:41.230936    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-gd6w9"
Jul 23 14:52:41.238748 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:41.238726    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-ns8qd"
Jul 23 14:52:41.246314 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:41.246296    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-68mqt"
Jul 23 14:52:41.252147 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:41.252129    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9qk7h"
Jul 23 14:52:41.261538 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:41.261514    2560 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 23 14:52:41.500898 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:41.500813    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/413d2464-7258-4246-85bb-ac9f03976ee5-metrics-certs\") pod \"network-metrics-daemon-hvw8v\" (UID: \"413d2464-7258-4246-85bb-ac9f03976ee5\") " pod="openshift-multus/network-metrics-daemon-hvw8v"
Jul 23 14:52:41.501063 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:52:41.500980    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 14:52:41.501063 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:52:41.501058    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/413d2464-7258-4246-85bb-ac9f03976ee5-metrics-certs podName:413d2464-7258-4246-85bb-ac9f03976ee5 nodeName:}" failed. No retries permitted until 2026-07-23 14:52:42.50103938 +0000 UTC m=+4.008913125 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/413d2464-7258-4246-85bb-ac9f03976ee5-metrics-certs") pod "network-metrics-daemon-hvw8v" (UID: "413d2464-7258-4246-85bb-ac9f03976ee5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 14:52:41.602065 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:41.602013    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/527775ad-7c13-4732-b824-4302ad79dcf4-original-pull-secret\") pod \"global-pull-secret-syncer-hvsnb\" (UID: \"527775ad-7c13-4732-b824-4302ad79dcf4\") " pod="kube-system/global-pull-secret-syncer-hvsnb"
Jul 23 14:52:41.602065 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:41.602062    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcqwm\" (UniqueName: \"kubernetes.io/projected/3370857a-3ffb-4603-bcad-503b48c10fd6-kube-api-access-fcqwm\") pod \"network-check-target-7sqtc\" (UID: \"3370857a-3ffb-4603-bcad-503b48c10fd6\") " pod="openshift-network-diagnostics/network-check-target-7sqtc"
Jul 23 14:52:41.602386 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:52:41.602197    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 23 14:52:41.602386 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:52:41.602249    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 23 14:52:41.602386 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:52:41.602273    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 23 14:52:41.602386 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:52:41.602287    2560 projected.go:196] Error preparing data for projected volume kube-api-access-fcqwm for pod openshift-network-diagnostics/network-check-target-7sqtc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 14:52:41.602386 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:52:41.602292    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/527775ad-7c13-4732-b824-4302ad79dcf4-original-pull-secret podName:527775ad-7c13-4732-b824-4302ad79dcf4 nodeName:}" failed. No retries permitted until 2026-07-23 14:52:42.602277312 +0000 UTC m=+4.110151043 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/527775ad-7c13-4732-b824-4302ad79dcf4-original-pull-secret") pod "global-pull-secret-syncer-hvsnb" (UID: "527775ad-7c13-4732-b824-4302ad79dcf4") : object "kube-system"/"original-pull-secret" not registered
Jul 23 14:52:41.602386 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:52:41.602337    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3370857a-3ffb-4603-bcad-503b48c10fd6-kube-api-access-fcqwm podName:3370857a-3ffb-4603-bcad-503b48c10fd6 nodeName:}" failed. No retries permitted until 2026-07-23 14:52:42.602321361 +0000 UTC m=+4.110195090 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-fcqwm" (UniqueName: "kubernetes.io/projected/3370857a-3ffb-4603-bcad-503b48c10fd6-kube-api-access-fcqwm") pod "network-check-target-7sqtc" (UID: "3370857a-3ffb-4603-bcad-503b48c10fd6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 14:52:41.775606 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:41.775533    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a2b87d6_d3dd_41b8_96f8_2e7a0be09325.slice/crio-6d560a346040be0848e20ca0e6b3c25319b691a8aab6990d464f02a62a7dbf38 WatchSource:0}: Error finding container 6d560a346040be0848e20ca0e6b3c25319b691a8aab6990d464f02a62a7dbf38: Status 404 returned error can't find the container with id 6d560a346040be0848e20ca0e6b3c25319b691a8aab6990d464f02a62a7dbf38
Jul 23 14:52:41.779208 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:41.779188    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fbaa452_6ab5_4370_ae18_fb4290ce9c10.slice/crio-9f3b66839da1f0a439b03977aed07b8f49e03aa5eb1c356daa971485309f14d0 WatchSource:0}: Error finding container 9f3b66839da1f0a439b03977aed07b8f49e03aa5eb1c356daa971485309f14d0: Status 404 returned error can't find the container with id 9f3b66839da1f0a439b03977aed07b8f49e03aa5eb1c356daa971485309f14d0
Jul 23 14:52:41.780626 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:41.780535    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4471b3d_89b5_46d0_b16d_734e52f04801.slice/crio-4e9ad6ecda06d6b061220b7e708626a65e5117d485cd81cc28b4a7484a95eb9c WatchSource:0}: Error finding container 4e9ad6ecda06d6b061220b7e708626a65e5117d485cd81cc28b4a7484a95eb9c: Status 404 returned error can't find the container with id 4e9ad6ecda06d6b061220b7e708626a65e5117d485cd81cc28b4a7484a95eb9c
Jul 23 14:52:41.781011 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:41.780997    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3df911b_3c63_441b_97fa_8d431bf3fb95.slice/crio-da2439e36c59a152f519c917d135400bfb692f28315b4e9d8d867ae859aa0461 WatchSource:0}: Error finding container da2439e36c59a152f519c917d135400bfb692f28315b4e9d8d867ae859aa0461: Status 404 returned error can't find the container with id da2439e36c59a152f519c917d135400bfb692f28315b4e9d8d867ae859aa0461
Jul 23 14:52:41.782012 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:41.781989    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37ed7912_d183_41e3_994b_561a8815f7d5.slice/crio-c12f11e5c1b6ed20b3cfbbf9168e44b1200a9c0544d5bc96a8c207b05f296940 WatchSource:0}: Error finding container c12f11e5c1b6ed20b3cfbbf9168e44b1200a9c0544d5bc96a8c207b05f296940: Status 404 returned error can't find the container with id c12f11e5c1b6ed20b3cfbbf9168e44b1200a9c0544d5bc96a8c207b05f296940
Jul 23 14:52:41.784258 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:41.783665    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc81888a_6094_4c2e_b881_6b9cd628e6c1.slice/crio-5541d83ff7af79e069e951cd0195cf214e56286be150559fa29b68c1117e03e6 WatchSource:0}: Error finding container 5541d83ff7af79e069e951cd0195cf214e56286be150559fa29b68c1117e03e6: Status 404 returned error can't find the container with id 5541d83ff7af79e069e951cd0195cf214e56286be150559fa29b68c1117e03e6
Jul 23 14:52:41.784558 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:41.784375    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95f95dc6_4b74_428b_8d51_d7975d5ed0e2.slice/crio-92bc374d99b51edf04dddf2c0389b84ef674efbf1041bfe331c01357438bdaea WatchSource:0}: Error finding container 92bc374d99b51edf04dddf2c0389b84ef674efbf1041bfe331c01357438bdaea: Status 404 returned error can't find the container with id 92bc374d99b51edf04dddf2c0389b84ef674efbf1041bfe331c01357438bdaea
Jul 23 14:52:41.785463 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:41.785371    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9b3550d_e82f_4934_b093_486563f240fb.slice/crio-26b1a06aaf63c13c410a82029a52f1d1314f35fe86665202c06f2c2cad3bbb10 WatchSource:0}: Error finding container 26b1a06aaf63c13c410a82029a52f1d1314f35fe86665202c06f2c2cad3bbb10: Status 404 returned error can't find the container with id 26b1a06aaf63c13c410a82029a52f1d1314f35fe86665202c06f2c2cad3bbb10
Jul 23 14:52:41.788393 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:52:41.788371    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0c66f26_06fe_4273_ba29_fa20ce89b873.slice/crio-2761c6bf809a115ef3ff8bc347cba5a92fce4b9345b4447b758ced9b3f21d548 WatchSource:0}: Error finding container 2761c6bf809a115ef3ff8bc347cba5a92fce4b9345b4447b758ced9b3f21d548: Status 404 returned error can't find the container with id 2761c6bf809a115ef3ff8bc347cba5a92fce4b9345b4447b758ced9b3f21d548
Jul 23 14:52:41.942519 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:41.942316    2560 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-22 14:47:40 +0000 UTC" deadline="2028-02-15 10:46:33.49168206 +0000 UTC"
Jul 23 14:52:41.942519 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:41.942516    2560 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13723h53m51.549169891s"
Jul 23 14:52:42.056824 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:42.056725    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7sqtc"
Jul 23 14:52:42.056824 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:42.056767    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hvw8v"
Jul 23 14:52:42.056967 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:52:42.056845    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7sqtc" podUID="3370857a-3ffb-4603-bcad-503b48c10fd6"
Jul 23 14:52:42.057015 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:52:42.056961    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hvw8v" podUID="413d2464-7258-4246-85bb-ac9f03976ee5"
Jul 23 14:52:42.064127 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:42.064094    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-vfsh6" event={"ID":"d9b3550d-e82f-4934-b093-486563f240fb","Type":"ContainerStarted","Data":"26b1a06aaf63c13c410a82029a52f1d1314f35fe86665202c06f2c2cad3bbb10"}
Jul 23 14:52:42.065196 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:42.065172    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9qk7h" event={"ID":"95f95dc6-4b74-428b-8d51-d7975d5ed0e2","Type":"ContainerStarted","Data":"92bc374d99b51edf04dddf2c0389b84ef674efbf1041bfe331c01357438bdaea"}
Jul 23 14:52:42.066276 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:42.066254    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-68mqt" event={"ID":"bc81888a-6094-4c2e-b881-6b9cd628e6c1","Type":"ContainerStarted","Data":"5541d83ff7af79e069e951cd0195cf214e56286be150559fa29b68c1117e03e6"}
Jul 23 14:52:42.067170 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:42.067141    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-lz8qd" event={"ID":"9a2b87d6-d3dd-41b8-96f8-2e7a0be09325","Type":"ContainerStarted","Data":"6d560a346040be0848e20ca0e6b3c25319b691a8aab6990d464f02a62a7dbf38"}
Jul 23 14:52:42.068187 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:42.068166    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z4fvn" event={"ID":"e0c66f26-06fe-4273-ba29-fa20ce89b873","Type":"ContainerStarted","Data":"2761c6bf809a115ef3ff8bc347cba5a92fce4b9345b4447b758ced9b3f21d548"}
Jul 23 14:52:42.069113 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:42.069094    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rmbz7" event={"ID":"37ed7912-d183-41e3-994b-561a8815f7d5","Type":"ContainerStarted","Data":"c12f11e5c1b6ed20b3cfbbf9168e44b1200a9c0544d5bc96a8c207b05f296940"}
Jul 23 14:52:42.070004 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:42.069986    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-gd6w9" event={"ID":"c4471b3d-89b5-46d0-b16d-734e52f04801","Type":"ContainerStarted","Data":"4e9ad6ecda06d6b061220b7e708626a65e5117d485cd81cc28b4a7484a95eb9c"}
Jul 23 14:52:42.070898 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:42.070878    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-ns8qd" event={"ID":"b3df911b-3c63-441b-97fa-8d431bf3fb95","Type":"ContainerStarted","Data":"da2439e36c59a152f519c917d135400bfb692f28315b4e9d8d867ae859aa0461"}
Jul 23 14:52:42.071860 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:42.071844    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q79jt" event={"ID":"6fbaa452-6ab5-4370-ae18-fb4290ce9c10","Type":"ContainerStarted","Data":"9f3b66839da1f0a439b03977aed07b8f49e03aa5eb1c356daa971485309f14d0"}
Jul 23 14:52:42.073613 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:42.073584    2560 generic.go:356] "Generic (PLEG): container finished" podID="666bf80ba1e1983d2d3dd851bc8ab130" containerID="9a2d9762f37da512742aeef782ab6daf1c34c24cedffbb630d6f5b192e1dfaff" exitCode=0
Jul 23 14:52:42.073613 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:42.073605    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-105.ec2.internal" event={"ID":"666bf80ba1e1983d2d3dd851bc8ab130","Type":"ContainerDied","Data":"9a2d9762f37da512742aeef782ab6daf1c34c24cedffbb630d6f5b192e1dfaff"}
Jul 23 14:52:42.074977 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:42.074960    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-105.ec2.internal" event={"ID":"3d37fd728dd468b7ee1c820042e75b2d","Type":"ContainerStarted","Data":"93708b649e1dea5c86a4cbdab4eb33a5584aeee57d9fb2a43e232b73b939031e"}
Jul 23 14:52:42.121024 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:42.120966    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-105.ec2.internal" podStartSLOduration=2.120949674 podStartE2EDuration="2.120949674s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 14:52:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 14:52:42.120430096 +0000 UTC m=+3.628303845" watchObservedRunningTime="2026-07-23 14:52:42.120949674 +0000 UTC m=+3.628823422"
Jul 23 14:52:42.509056 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:42.508981    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/413d2464-7258-4246-85bb-ac9f03976ee5-metrics-certs\") pod \"network-metrics-daemon-hvw8v\" (UID: \"413d2464-7258-4246-85bb-ac9f03976ee5\") " pod="openshift-multus/network-metrics-daemon-hvw8v"
Jul 23 14:52:42.509215 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:52:42.509145    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 14:52:42.509303 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:52:42.509216    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/413d2464-7258-4246-85bb-ac9f03976ee5-metrics-certs podName:413d2464-7258-4246-85bb-ac9f03976ee5 nodeName:}" failed. No retries permitted until 2026-07-23 14:52:44.509194196 +0000 UTC m=+6.017067935 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/413d2464-7258-4246-85bb-ac9f03976ee5-metrics-certs") pod "network-metrics-daemon-hvw8v" (UID: "413d2464-7258-4246-85bb-ac9f03976ee5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 14:52:42.610264 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:42.610209    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/527775ad-7c13-4732-b824-4302ad79dcf4-original-pull-secret\") pod \"global-pull-secret-syncer-hvsnb\" (UID: \"527775ad-7c13-4732-b824-4302ad79dcf4\") " pod="kube-system/global-pull-secret-syncer-hvsnb"
Jul 23 14:52:42.610444 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:42.610279    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcqwm\" (UniqueName: \"kubernetes.io/projected/3370857a-3ffb-4603-bcad-503b48c10fd6-kube-api-access-fcqwm\") pod \"network-check-target-7sqtc\" (UID: \"3370857a-3ffb-4603-bcad-503b48c10fd6\") " pod="openshift-network-diagnostics/network-check-target-7sqtc"
Jul 23 14:52:42.610511 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:52:42.610442    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 23 14:52:42.610511 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:52:42.610461    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 23 14:52:42.610511 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:52:42.610474    2560 projected.go:196] Error preparing data for projected volume kube-api-access-fcqwm for pod openshift-network-diagnostics/network-check-target-7sqtc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 14:52:42.610663 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:52:42.610531    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3370857a-3ffb-4603-bcad-503b48c10fd6-kube-api-access-fcqwm podName:3370857a-3ffb-4603-bcad-503b48c10fd6 nodeName:}" failed. No retries permitted until 2026-07-23 14:52:44.610515015 +0000 UTC m=+6.118388746 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-fcqwm" (UniqueName: "kubernetes.io/projected/3370857a-3ffb-4603-bcad-503b48c10fd6-kube-api-access-fcqwm") pod "network-check-target-7sqtc" (UID: "3370857a-3ffb-4603-bcad-503b48c10fd6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 14:52:42.611014 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:52:42.610954    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 23 14:52:42.611014 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:52:42.611008    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/527775ad-7c13-4732-b824-4302ad79dcf4-original-pull-secret podName:527775ad-7c13-4732-b824-4302ad79dcf4 nodeName:}" failed. No retries permitted until 2026-07-23 14:52:44.61099331 +0000 UTC m=+6.118867052 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/527775ad-7c13-4732-b824-4302ad79dcf4-original-pull-secret") pod "global-pull-secret-syncer-hvsnb" (UID: "527775ad-7c13-4732-b824-4302ad79dcf4") : object "kube-system"/"original-pull-secret" not registered
Jul 23 14:52:42.943323 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:42.942862    2560 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-22 14:47:40 +0000 UTC" deadline="2028-03-12 04:13:42.427679378 +0000 UTC"
Jul 23 14:52:42.943323 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:42.942899    2560 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14341h20m59.484787303s"
Jul 23 14:52:43.060589 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:43.059355    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hvsnb"
Jul 23 14:52:43.060589 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:52:43.059548    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hvsnb" podUID="527775ad-7c13-4732-b824-4302ad79dcf4"
Jul 23 14:52:43.094783 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:43.094742    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-105.ec2.internal" event={"ID":"666bf80ba1e1983d2d3dd851bc8ab130","Type":"ContainerStarted","Data":"4fa70a98980867d5476205d6f5a34faf71a1a61fc669fb311602d9a34b6296bc"}
Jul 23 14:52:43.135628 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:43.135540    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-105.ec2.internal" podStartSLOduration=3.135518081 podStartE2EDuration="3.135518081s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 14:52:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 14:52:43.13362782 +0000 UTC m=+4.641501568" watchObservedRunningTime="2026-07-23 14:52:43.135518081 +0000 UTC m=+4.643391830"
Jul 23 14:52:44.057829 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:44.056998    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hvw8v"
Jul 23 14:52:44.057829 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:52:44.057200    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hvw8v" podUID="413d2464-7258-4246-85bb-ac9f03976ee5"
Jul 23 14:52:44.057829 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:44.057311    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7sqtc"
Jul 23 14:52:44.057829 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:52:44.057389    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7sqtc" podUID="3370857a-3ffb-4603-bcad-503b48c10fd6"
Jul 23 14:52:44.526369 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:44.526268    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/413d2464-7258-4246-85bb-ac9f03976ee5-metrics-certs\") pod \"network-metrics-daemon-hvw8v\" (UID: \"413d2464-7258-4246-85bb-ac9f03976ee5\") " pod="openshift-multus/network-metrics-daemon-hvw8v"
Jul 23 14:52:44.526532 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:52:44.526481    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 14:52:44.526600 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:52:44.526552    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/413d2464-7258-4246-85bb-ac9f03976ee5-metrics-certs podName:413d2464-7258-4246-85bb-ac9f03976ee5 nodeName:}" failed. No retries permitted until 2026-07-23 14:52:48.526531164 +0000 UTC m=+10.034404894 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/413d2464-7258-4246-85bb-ac9f03976ee5-metrics-certs") pod "network-metrics-daemon-hvw8v" (UID: "413d2464-7258-4246-85bb-ac9f03976ee5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 14:52:44.627273 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:44.627218    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/527775ad-7c13-4732-b824-4302ad79dcf4-original-pull-secret\") pod \"global-pull-secret-syncer-hvsnb\" (UID: \"527775ad-7c13-4732-b824-4302ad79dcf4\") " pod="kube-system/global-pull-secret-syncer-hvsnb"
Jul 23 14:52:44.627441 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:44.627293    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcqwm\" (UniqueName: \"kubernetes.io/projected/3370857a-3ffb-4603-bcad-503b48c10fd6-kube-api-access-fcqwm\") pod \"network-check-target-7sqtc\" (UID: \"3370857a-3ffb-4603-bcad-503b48c10fd6\") " pod="openshift-network-diagnostics/network-check-target-7sqtc"
Jul 23 14:52:44.627528 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:52:44.627498    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 23 14:52:44.627528 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:52:44.627519    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 23 14:52:44.627630 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:52:44.627533    2560 projected.go:196] Error preparing data for projected volume kube-api-access-fcqwm for pod openshift-network-diagnostics/network-check-target-7sqtc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 14:52:44.627630 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:52:44.627585    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3370857a-3ffb-4603-bcad-503b48c10fd6-kube-api-access-fcqwm podName:3370857a-3ffb-4603-bcad-503b48c10fd6 nodeName:}" failed. No retries permitted until 2026-07-23 14:52:48.627571927 +0000 UTC m=+10.135445653 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-fcqwm" (UniqueName: "kubernetes.io/projected/3370857a-3ffb-4603-bcad-503b48c10fd6-kube-api-access-fcqwm") pod "network-check-target-7sqtc" (UID: "3370857a-3ffb-4603-bcad-503b48c10fd6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 14:52:44.627985 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:52:44.627959    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 23 14:52:44.628060 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:52:44.628016    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/527775ad-7c13-4732-b824-4302ad79dcf4-original-pull-secret podName:527775ad-7c13-4732-b824-4302ad79dcf4 nodeName:}" failed. No retries permitted until 2026-07-23 14:52:48.628001415 +0000 UTC m=+10.135875145 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/527775ad-7c13-4732-b824-4302ad79dcf4-original-pull-secret") pod "global-pull-secret-syncer-hvsnb" (UID: "527775ad-7c13-4732-b824-4302ad79dcf4") : object "kube-system"/"original-pull-secret" not registered
Jul 23 14:52:45.057129 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:45.056741    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hvsnb"
Jul 23 14:52:45.057129 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:52:45.056857    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hvsnb" podUID="527775ad-7c13-4732-b824-4302ad79dcf4"
Jul 23 14:52:46.058085 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:46.057390    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7sqtc"
Jul 23 14:52:46.058085 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:52:46.057547    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7sqtc" podUID="3370857a-3ffb-4603-bcad-503b48c10fd6"
Jul 23 14:52:46.058085 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:46.057938    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hvw8v"
Jul 23 14:52:46.058085 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:52:46.058039    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hvw8v" podUID="413d2464-7258-4246-85bb-ac9f03976ee5"
Jul 23 14:52:47.059004 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:47.058972    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hvsnb"
Jul 23 14:52:47.059466 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:52:47.059111    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hvsnb" podUID="527775ad-7c13-4732-b824-4302ad79dcf4"
Jul 23 14:52:48.057442 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:48.057410    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hvw8v"
Jul 23 14:52:48.057623 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:48.057414    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7sqtc"
Jul 23 14:52:48.057623 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:52:48.057566    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hvw8v" podUID="413d2464-7258-4246-85bb-ac9f03976ee5"
Jul 23 14:52:48.057947 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:52:48.057921    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7sqtc" podUID="3370857a-3ffb-4603-bcad-503b48c10fd6"
Jul 23 14:52:48.564250 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:48.564202    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/413d2464-7258-4246-85bb-ac9f03976ee5-metrics-certs\") pod \"network-metrics-daemon-hvw8v\" (UID: \"413d2464-7258-4246-85bb-ac9f03976ee5\") " pod="openshift-multus/network-metrics-daemon-hvw8v"
Jul 23 14:52:48.564698 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:52:48.564384    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 14:52:48.564698 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:52:48.564451    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/413d2464-7258-4246-85bb-ac9f03976ee5-metrics-certs podName:413d2464-7258-4246-85bb-ac9f03976ee5 nodeName:}" failed. No retries permitted until 2026-07-23 14:52:56.564431942 +0000 UTC m=+18.072305682 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/413d2464-7258-4246-85bb-ac9f03976ee5-metrics-certs") pod "network-metrics-daemon-hvw8v" (UID: "413d2464-7258-4246-85bb-ac9f03976ee5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 14:52:48.666047 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:48.665347    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/527775ad-7c13-4732-b824-4302ad79dcf4-original-pull-secret\") pod \"global-pull-secret-syncer-hvsnb\" (UID: \"527775ad-7c13-4732-b824-4302ad79dcf4\") " pod="kube-system/global-pull-secret-syncer-hvsnb"
Jul 23 14:52:48.666047 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:48.665397    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcqwm\" (UniqueName: \"kubernetes.io/projected/3370857a-3ffb-4603-bcad-503b48c10fd6-kube-api-access-fcqwm\") pod \"network-check-target-7sqtc\" (UID: \"3370857a-3ffb-4603-bcad-503b48c10fd6\") " pod="openshift-network-diagnostics/network-check-target-7sqtc"
Jul 23 14:52:48.666047 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:52:48.665541    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 23 14:52:48.666047 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:52:48.665556    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 23 14:52:48.666047 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:52:48.665564    2560 projected.go:196] Error preparing data for projected volume kube-api-access-fcqwm for pod openshift-network-diagnostics/network-check-target-7sqtc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 14:52:48.666047 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:52:48.665607    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3370857a-3ffb-4603-bcad-503b48c10fd6-kube-api-access-fcqwm podName:3370857a-3ffb-4603-bcad-503b48c10fd6 nodeName:}" failed. No retries permitted until 2026-07-23 14:52:56.665594804 +0000 UTC m=+18.173468532 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-fcqwm" (UniqueName: "kubernetes.io/projected/3370857a-3ffb-4603-bcad-503b48c10fd6-kube-api-access-fcqwm") pod "network-check-target-7sqtc" (UID: "3370857a-3ffb-4603-bcad-503b48c10fd6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 14:52:48.666047 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:52:48.665965    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 23 14:52:48.666047 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:52:48.666015    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/527775ad-7c13-4732-b824-4302ad79dcf4-original-pull-secret podName:527775ad-7c13-4732-b824-4302ad79dcf4 nodeName:}" failed. No retries permitted until 2026-07-23 14:52:56.665999145 +0000 UTC m=+18.173872876 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/527775ad-7c13-4732-b824-4302ad79dcf4-original-pull-secret") pod "global-pull-secret-syncer-hvsnb" (UID: "527775ad-7c13-4732-b824-4302ad79dcf4") : object "kube-system"/"original-pull-secret" not registered
Jul 23 14:52:49.059260 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:49.059211    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hvsnb"
Jul 23 14:52:49.059439 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:52:49.059354    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hvsnb" podUID="527775ad-7c13-4732-b824-4302ad79dcf4"
Jul 23 14:52:50.056941 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:50.056898    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hvw8v"
Jul 23 14:52:50.057383 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:52:50.057032    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hvw8v" podUID="413d2464-7258-4246-85bb-ac9f03976ee5"
Jul 23 14:52:50.057455 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:50.057384    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7sqtc"
Jul 23 14:52:50.057455 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:52:50.057449    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7sqtc" podUID="3370857a-3ffb-4603-bcad-503b48c10fd6"
Jul 23 14:52:51.057339 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:51.057308    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hvsnb"
Jul 23 14:52:51.057777 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:52:51.057444    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hvsnb" podUID="527775ad-7c13-4732-b824-4302ad79dcf4"
Jul 23 14:52:52.056930 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:52.056895    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hvw8v"
Jul 23 14:52:52.056930 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:52.056926    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7sqtc"
Jul 23 14:52:52.057170 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:52:52.057024    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hvw8v" podUID="413d2464-7258-4246-85bb-ac9f03976ee5"
Jul 23 14:52:52.057170 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:52:52.057135    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7sqtc" podUID="3370857a-3ffb-4603-bcad-503b48c10fd6"
Jul 23 14:52:53.056874 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:53.056838    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hvsnb"
Jul 23 14:52:53.057295 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:52:53.056968    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hvsnb" podUID="527775ad-7c13-4732-b824-4302ad79dcf4"
Jul 23 14:52:54.056599 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:54.056556    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hvw8v"
Jul 23 14:52:54.056782 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:54.056602    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7sqtc"
Jul 23 14:52:54.056782 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:52:54.056707    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hvw8v" podUID="413d2464-7258-4246-85bb-ac9f03976ee5"
Jul 23 14:52:54.056901 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:52:54.056858    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7sqtc" podUID="3370857a-3ffb-4603-bcad-503b48c10fd6"
Jul 23 14:52:55.056863 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:55.056825    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hvsnb"
Jul 23 14:52:55.057024 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:52:55.056975    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hvsnb" podUID="527775ad-7c13-4732-b824-4302ad79dcf4"
Jul 23 14:52:56.056513 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:56.056483    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hvw8v"
Jul 23 14:52:56.056677 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:52:56.056589    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hvw8v" podUID="413d2464-7258-4246-85bb-ac9f03976ee5"
Jul 23 14:52:56.056677 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:56.056658    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7sqtc"
Jul 23 14:52:56.056796 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:52:56.056768    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7sqtc" podUID="3370857a-3ffb-4603-bcad-503b48c10fd6"
Jul 23 14:52:56.623568 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:56.623521    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/413d2464-7258-4246-85bb-ac9f03976ee5-metrics-certs\") pod \"network-metrics-daemon-hvw8v\" (UID: \"413d2464-7258-4246-85bb-ac9f03976ee5\") " pod="openshift-multus/network-metrics-daemon-hvw8v"
Jul 23 14:52:56.624017 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:52:56.623666    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 14:52:56.624017 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:52:56.623735    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/413d2464-7258-4246-85bb-ac9f03976ee5-metrics-certs podName:413d2464-7258-4246-85bb-ac9f03976ee5 nodeName:}" failed. No retries permitted until 2026-07-23 14:53:12.623712909 +0000 UTC m=+34.131586657 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/413d2464-7258-4246-85bb-ac9f03976ee5-metrics-certs") pod "network-metrics-daemon-hvw8v" (UID: "413d2464-7258-4246-85bb-ac9f03976ee5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 14:52:56.724344 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:56.724312    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcqwm\" (UniqueName: \"kubernetes.io/projected/3370857a-3ffb-4603-bcad-503b48c10fd6-kube-api-access-fcqwm\") pod \"network-check-target-7sqtc\" (UID: \"3370857a-3ffb-4603-bcad-503b48c10fd6\") " pod="openshift-network-diagnostics/network-check-target-7sqtc"
Jul 23 14:52:56.724537 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:56.724404    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/527775ad-7c13-4732-b824-4302ad79dcf4-original-pull-secret\") pod \"global-pull-secret-syncer-hvsnb\" (UID: \"527775ad-7c13-4732-b824-4302ad79dcf4\") " pod="kube-system/global-pull-secret-syncer-hvsnb"
Jul 23 14:52:56.724537 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:52:56.724518    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 23 14:52:56.724700 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:52:56.724581    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/527775ad-7c13-4732-b824-4302ad79dcf4-original-pull-secret podName:527775ad-7c13-4732-b824-4302ad79dcf4 nodeName:}" failed. No retries permitted until 2026-07-23 14:53:12.724563188 +0000 UTC m=+34.232436928 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/527775ad-7c13-4732-b824-4302ad79dcf4-original-pull-secret") pod "global-pull-secret-syncer-hvsnb" (UID: "527775ad-7c13-4732-b824-4302ad79dcf4") : object "kube-system"/"original-pull-secret" not registered
Jul 23 14:52:56.724892 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:52:56.724869    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 23 14:52:56.724892 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:52:56.724891    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 23 14:52:56.725067 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:52:56.724904    2560 projected.go:196] Error preparing data for projected volume kube-api-access-fcqwm for pod openshift-network-diagnostics/network-check-target-7sqtc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 14:52:56.725067 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:52:56.724949    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3370857a-3ffb-4603-bcad-503b48c10fd6-kube-api-access-fcqwm podName:3370857a-3ffb-4603-bcad-503b48c10fd6 nodeName:}" failed. No retries permitted until 2026-07-23 14:53:12.724936161 +0000 UTC m=+34.232809889 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-fcqwm" (UniqueName: "kubernetes.io/projected/3370857a-3ffb-4603-bcad-503b48c10fd6-kube-api-access-fcqwm") pod "network-check-target-7sqtc" (UID: "3370857a-3ffb-4603-bcad-503b48c10fd6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 14:52:57.057371 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:57.057324    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hvsnb"
Jul 23 14:52:57.057556 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:52:57.057462    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hvsnb" podUID="527775ad-7c13-4732-b824-4302ad79dcf4"
Jul 23 14:52:58.057015 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:58.056976    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hvw8v"
Jul 23 14:52:58.057461 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:52:58.057115    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hvw8v" podUID="413d2464-7258-4246-85bb-ac9f03976ee5"
Jul 23 14:52:58.057461 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:58.057173    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7sqtc"
Jul 23 14:52:58.057461 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:52:58.057276    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7sqtc" podUID="3370857a-3ffb-4603-bcad-503b48c10fd6"
Jul 23 14:52:59.058200 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:52:59.058159    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hvsnb"
Jul 23 14:52:59.058664 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:52:59.058299    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hvsnb" podUID="527775ad-7c13-4732-b824-4302ad79dcf4"
Jul 23 14:53:00.057153 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:00.057118    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hvw8v"
Jul 23 14:53:00.057370 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:00.057167    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7sqtc"
Jul 23 14:53:00.057370 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:53:00.057278    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hvw8v" podUID="413d2464-7258-4246-85bb-ac9f03976ee5"
Jul 23 14:53:00.057485 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:53:00.057392    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7sqtc" podUID="3370857a-3ffb-4603-bcad-503b48c10fd6"
Jul 23 14:53:01.057806 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:01.057625    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hvsnb"
Jul 23 14:53:01.058606 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:53:01.057882    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hvsnb" podUID="527775ad-7c13-4732-b824-4302ad79dcf4"
Jul 23 14:53:01.136504 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:01.136390    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z4fvn_e0c66f26-06fe-4273-ba29-fa20ce89b873/ovn-acl-logging/0.log"
Jul 23 14:53:01.136836 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:01.136814    2560 generic.go:356] "Generic (PLEG): container finished" podID="e0c66f26-06fe-4273-ba29-fa20ce89b873" containerID="aa61e743815c0645e2425969537e7a3f2f0f6d847cd9bcced4f49c5fe91b5a65" exitCode=1
Jul 23 14:53:01.136915 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:01.136840    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z4fvn" event={"ID":"e0c66f26-06fe-4273-ba29-fa20ce89b873","Type":"ContainerStarted","Data":"89b03f78a83d6685caf5e608cd97ef1fe28ecc2b81d206cf19b5bce585d6ac1f"}
Jul 23 14:53:01.136965 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:01.136916    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z4fvn" event={"ID":"e0c66f26-06fe-4273-ba29-fa20ce89b873","Type":"ContainerDied","Data":"aa61e743815c0645e2425969537e7a3f2f0f6d847cd9bcced4f49c5fe91b5a65"}
Jul 23 14:53:01.137009 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:01.136967    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z4fvn" event={"ID":"e0c66f26-06fe-4273-ba29-fa20ce89b873","Type":"ContainerStarted","Data":"abba3386eb68de713d9a0fd05baa9736c170bd74cb61d2d0154e34da20644b22"}
Jul 23 14:53:01.138306 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:01.138285    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rmbz7" event={"ID":"37ed7912-d183-41e3-994b-561a8815f7d5","Type":"ContainerStarted","Data":"3a9c0a6d9bdde8218097532056bfb1a356eb1b92d775656bd33f4f171d3d0e43"}
Jul 23 14:53:01.139583 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:01.139556    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-ns8qd" event={"ID":"b3df911b-3c63-441b-97fa-8d431bf3fb95","Type":"ContainerStarted","Data":"1b4c8410e7f53c74865029c8ebf77aeeed013e065f7e294170a35bc40b6a5eda"}
Jul 23 14:53:01.140764 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:01.140740    2560 generic.go:356] "Generic (PLEG): container finished" podID="6fbaa452-6ab5-4370-ae18-fb4290ce9c10" containerID="1189423a10e88c2ff7e399b505fd463c8f8ba7e69205498d270c1818f823865a" exitCode=0
Jul 23 14:53:01.140841 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:01.140804    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q79jt" event={"ID":"6fbaa452-6ab5-4370-ae18-fb4290ce9c10","Type":"ContainerDied","Data":"1189423a10e88c2ff7e399b505fd463c8f8ba7e69205498d270c1818f823865a"}
Jul 23 14:53:01.142828 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:01.142618    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-vfsh6" event={"ID":"d9b3550d-e82f-4934-b093-486563f240fb","Type":"ContainerStarted","Data":"4aa19aee8230e4832e2c88fc7b78a99ef30935b2b5d5a67c452450208f0d5814"}
Jul 23 14:53:01.144352 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:01.144325    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9qk7h" event={"ID":"95f95dc6-4b74-428b-8d51-d7975d5ed0e2","Type":"ContainerStarted","Data":"052e9d26604a8cfae1d2e86274147699114926734dce5a30ff27553c6e17b8ef"}
Jul 23 14:53:01.145786 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:01.145768    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-68mqt" event={"ID":"bc81888a-6094-4c2e-b881-6b9cd628e6c1","Type":"ContainerStarted","Data":"f398fdc0d60ba6a1fa7e5dead32594d504e11c5a04637e9d125afb0c5ea96895"}
Jul 23 14:53:01.147074 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:01.147056    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-lz8qd" event={"ID":"9a2b87d6-d3dd-41b8-96f8-2e7a0be09325","Type":"ContainerStarted","Data":"5a2bb480acf391b4c7dcb6c8db76c9a68743cfb75a14fa29523d3bc98a467da4"}
Jul 23 14:53:01.177140 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:01.177099    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-ns8qd" podStartSLOduration=8.069892556 podStartE2EDuration="22.177084345s" totalImagesPullingTime="14.107191789s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 14:52:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 14:53:01.176936975 +0000 UTC m=+22.684810724" watchObservedRunningTime="2026-07-23 14:53:01.177084345 +0000 UTC m=+22.684958092"
Jul 23 14:53:01.177413 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:01.177392    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-rmbz7" podStartSLOduration=3.285058482 podStartE2EDuration="22.177385329s" totalImagesPullingTime="18.892326847s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 14:52:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 14:53:01.160779627 +0000 UTC m=+22.668653375" watchObservedRunningTime="2026-07-23 14:53:01.177385329 +0000 UTC m=+22.685259077"
Jul 23 14:53:01.194182 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:01.194136    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-68mqt" podStartSLOduration=3.269265749 podStartE2EDuration="22.194121433s" totalImagesPullingTime="18.924855684s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 14:52:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 14:53:01.193751336 +0000 UTC m=+22.701625083" watchObservedRunningTime="2026-07-23 14:53:01.194121433 +0000 UTC m=+22.701995180"
Jul 23 14:53:01.230465 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:01.230423    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-vfsh6" podStartSLOduration=3.344637282 podStartE2EDuration="22.230408082s" totalImagesPullingTime="18.8857708s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 14:52:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 14:53:01.230003645 +0000 UTC m=+22.737877396" watchObservedRunningTime="2026-07-23 14:53:01.230408082 +0000 UTC m=+22.738281829"
Jul 23 14:53:01.250425 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:01.250360    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-lz8qd" podStartSLOduration=8.137296974 podStartE2EDuration="22.250345031s" totalImagesPullingTime="14.113048057s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 14:52:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 14:53:01.250214858 +0000 UTC m=+22.758088607" watchObservedRunningTime="2026-07-23 14:53:01.250345031 +0000 UTC m=+22.758218778"
Jul 23 14:53:02.004998 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:02.004963    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-ns8qd"
Jul 23 14:53:02.005602 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:02.005583    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-ns8qd"
Jul 23 14:53:02.057249 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:02.057205    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hvw8v"
Jul 23 14:53:02.057403 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:02.057255    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7sqtc"
Jul 23 14:53:02.057403 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:53:02.057335    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hvw8v" podUID="413d2464-7258-4246-85bb-ac9f03976ee5"
Jul 23 14:53:02.057515 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:53:02.057440    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7sqtc" podUID="3370857a-3ffb-4603-bcad-503b48c10fd6"
Jul 23 14:53:02.152902 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:02.152879    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z4fvn_e0c66f26-06fe-4273-ba29-fa20ce89b873/ovn-acl-logging/0.log"
Jul 23 14:53:02.153397 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:02.153296    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z4fvn" event={"ID":"e0c66f26-06fe-4273-ba29-fa20ce89b873","Type":"ContainerStarted","Data":"e0e89c28780456df6d657467560b3bca6fb1d150a88cfcf420ea3e1d386d2b70"}
Jul 23 14:53:02.153397 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:02.153358    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z4fvn" event={"ID":"e0c66f26-06fe-4273-ba29-fa20ce89b873","Type":"ContainerStarted","Data":"f3c83ca6099f2bbd7655e989416697e3086c337374b005f476281c29d433bac3"}
Jul 23 14:53:02.153489 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:02.153402    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z4fvn" event={"ID":"e0c66f26-06fe-4273-ba29-fa20ce89b873","Type":"ContainerStarted","Data":"599dbd36634c46be38efb1d615ab16ce788ac28f4955e0587fbc6443c2a87e51"}
Jul 23 14:53:02.154794 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:02.154771    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-gd6w9" event={"ID":"c4471b3d-89b5-46d0-b16d-734e52f04801","Type":"ContainerStarted","Data":"93a0c72ce9bb600dc23aab8802791d338cb0567169d83adc03c0fa28759cd861"}
Jul 23 14:53:02.158472 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:02.158440    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-ns8qd"
Jul 23 14:53:02.158931 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:02.158912    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-ns8qd"
Jul 23 14:53:02.174984 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:02.174927    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-gd6w9" podStartSLOduration=4.280103297 podStartE2EDuration="23.174905447s" totalImagesPullingTime="18.89480215s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 14:52:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 14:53:02.17414868 +0000 UTC m=+23.682022429" watchObservedRunningTime="2026-07-23 14:53:02.174905447 +0000 UTC m=+23.682779197"
Jul 23 14:53:02.270778 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:02.270740    2560 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 23 14:53:02.958958 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:02.958824    2560 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-23T14:53:02.270760644Z","UUID":"2afa348c-83ba-41ab-8ee7-5e490e18ce05","Handler":null,"Name":"","Endpoint":""}
Jul 23 14:53:02.960708 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:02.960685    2560 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 23 14:53:02.960842 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:02.960731    2560 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 23 14:53:03.057494 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:03.057466    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hvsnb"
Jul 23 14:53:03.057660 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:53:03.057591    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hvsnb" podUID="527775ad-7c13-4732-b824-4302ad79dcf4"
Jul 23 14:53:03.160567 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:03.160532    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9qk7h" event={"ID":"95f95dc6-4b74-428b-8d51-d7975d5ed0e2","Type":"ContainerStarted","Data":"c57194a0fc65448be58ea5f39ecff796e48826363f5dc2565a3a04287ac7f8be"}
Jul 23 14:53:04.056660 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:04.056621    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7sqtc"
Jul 23 14:53:04.056842 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:04.056630    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hvw8v"
Jul 23 14:53:04.056842 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:53:04.056744    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7sqtc" podUID="3370857a-3ffb-4603-bcad-503b48c10fd6"
Jul 23 14:53:04.056955 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:53:04.056856    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hvw8v" podUID="413d2464-7258-4246-85bb-ac9f03976ee5"
Jul 23 14:53:04.165852 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:04.165828    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z4fvn_e0c66f26-06fe-4273-ba29-fa20ce89b873/ovn-acl-logging/0.log"
Jul 23 14:53:04.166246 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:04.166201    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z4fvn" event={"ID":"e0c66f26-06fe-4273-ba29-fa20ce89b873","Type":"ContainerStarted","Data":"3ee07d8fa9b03b2a385fbd8337b6be4a9dca008cdd2c24b91b5148531bee72ae"}
Jul 23 14:53:04.168424 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:04.168392    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9qk7h" event={"ID":"95f95dc6-4b74-428b-8d51-d7975d5ed0e2","Type":"ContainerStarted","Data":"47f79bd17c17a43a026c35a770ea4e6a53002aacb0c127d4643d13912aacf360"}
Jul 23 14:53:04.189155 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:04.189105    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9qk7h" podStartSLOduration=3.984308227 podStartE2EDuration="25.18909053s" totalImagesPullingTime="21.204782303s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 14:52:39 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 14:53:04.187089994 +0000 UTC m=+25.694963754" watchObservedRunningTime="2026-07-23 14:53:04.18909053 +0000 UTC m=+25.696964278"
Jul 23 14:53:05.060910 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:05.060728    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hvsnb"
Jul 23 14:53:05.061077 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:53:05.060994    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hvsnb" podUID="527775ad-7c13-4732-b824-4302ad79dcf4"
Jul 23 14:53:06.057191 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:06.057099    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7sqtc"
Jul 23 14:53:06.058042 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:53:06.057246    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7sqtc" podUID="3370857a-3ffb-4603-bcad-503b48c10fd6"
Jul 23 14:53:06.058042 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:06.057284    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hvw8v"
Jul 23 14:53:06.058042 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:53:06.057381    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hvw8v" podUID="413d2464-7258-4246-85bb-ac9f03976ee5"
Jul 23 14:53:06.175680 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:06.175655    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z4fvn_e0c66f26-06fe-4273-ba29-fa20ce89b873/ovn-acl-logging/0.log"
Jul 23 14:53:06.176052 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:06.176026    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z4fvn" event={"ID":"e0c66f26-06fe-4273-ba29-fa20ce89b873","Type":"ContainerStarted","Data":"00b955f83855735867e40eb5409307901ea8b43c4734edadb68cf88595bdb974"}
Jul 23 14:53:06.176309 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:06.176287    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-z4fvn"
Jul 23 14:53:06.176506 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:06.176489    2560 scope.go:117] "RemoveContainer" containerID="aa61e743815c0645e2425969537e7a3f2f0f6d847cd9bcced4f49c5fe91b5a65"
Jul 23 14:53:06.178353 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:06.178316    2560 generic.go:356] "Generic (PLEG): container finished" podID="6fbaa452-6ab5-4370-ae18-fb4290ce9c10" containerID="6ea19ac5baffe3e274a8f436f6d969b03a0286f7021e4a3c928bb3a268f67b46" exitCode=0
Jul 23 14:53:06.178473 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:06.178372    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q79jt" event={"ID":"6fbaa452-6ab5-4370-ae18-fb4290ce9c10","Type":"ContainerDied","Data":"6ea19ac5baffe3e274a8f436f6d969b03a0286f7021e4a3c928bb3a268f67b46"}
Jul 23 14:53:06.194248 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:06.194200    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-z4fvn"
Jul 23 14:53:07.056929 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:07.056737    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hvsnb"
Jul 23 14:53:07.057094 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:53:07.057018    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hvsnb" podUID="527775ad-7c13-4732-b824-4302ad79dcf4"
Jul 23 14:53:07.185493 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:07.185395    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z4fvn_e0c66f26-06fe-4273-ba29-fa20ce89b873/ovn-acl-logging/0.log"
Jul 23 14:53:07.185863 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:07.185816    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z4fvn" event={"ID":"e0c66f26-06fe-4273-ba29-fa20ce89b873","Type":"ContainerStarted","Data":"bbb52907544cf09df41c9a055f6e1bcb3dec796f0bf7fa96c8cdc6515a7c231a"}
Jul 23 14:53:07.186148 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:07.185979    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-z4fvn"
Jul 23 14:53:07.186148 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:07.186040    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-z4fvn"
Jul 23 14:53:07.188439 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:07.188409    2560 generic.go:356] "Generic (PLEG): container finished" podID="6fbaa452-6ab5-4370-ae18-fb4290ce9c10" containerID="e471dfb8dae62aba9dcf3c63e3f2176261a8585f4ddec938f0f40783905731c1" exitCode=0
Jul 23 14:53:07.188547 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:07.188454    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q79jt" event={"ID":"6fbaa452-6ab5-4370-ae18-fb4290ce9c10","Type":"ContainerDied","Data":"e471dfb8dae62aba9dcf3c63e3f2176261a8585f4ddec938f0f40783905731c1"}
Jul 23 14:53:07.203508 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:07.203470    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-z4fvn"
Jul 23 14:53:07.223655 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:07.221973    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-z4fvn" podStartSLOduration=9.267354689 podStartE2EDuration="28.221952663s" totalImagesPullingTime="18.954597974s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 14:52:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 14:53:07.219368722 +0000 UTC m=+28.727242467" watchObservedRunningTime="2026-07-23 14:53:07.221952663 +0000 UTC m=+28.729826412"
Jul 23 14:53:07.679461 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:07.679426    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-hvsnb"]
Jul 23 14:53:07.679689 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:07.679626    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hvsnb"
Jul 23 14:53:07.679753 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:53:07.679723    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hvsnb" podUID="527775ad-7c13-4732-b824-4302ad79dcf4"
Jul 23 14:53:07.681884 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:07.681860    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-7sqtc"]
Jul 23 14:53:07.682001 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:07.681971    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7sqtc"
Jul 23 14:53:07.682057 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:53:07.682042    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7sqtc" podUID="3370857a-3ffb-4603-bcad-503b48c10fd6"
Jul 23 14:53:07.695396 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:07.695290    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-hvw8v"]
Jul 23 14:53:07.695546 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:07.695436    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hvw8v"
Jul 23 14:53:07.695609 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:53:07.695542    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hvw8v" podUID="413d2464-7258-4246-85bb-ac9f03976ee5"
Jul 23 14:53:08.192849 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:08.192816    2560 generic.go:356] "Generic (PLEG): container finished" podID="6fbaa452-6ab5-4370-ae18-fb4290ce9c10" containerID="bc6fc5ce028edc5f4b17589a9694282bf85fe9580643f24af8480809f5bfd17f" exitCode=0
Jul 23 14:53:08.193344 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:08.192905    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q79jt" event={"ID":"6fbaa452-6ab5-4370-ae18-fb4290ce9c10","Type":"ContainerDied","Data":"bc6fc5ce028edc5f4b17589a9694282bf85fe9580643f24af8480809f5bfd17f"}
Jul 23 14:53:09.057423 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:09.057393    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7sqtc"
Jul 23 14:53:09.057591 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:53:09.057487    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7sqtc" podUID="3370857a-3ffb-4603-bcad-503b48c10fd6"
Jul 23 14:53:09.057740 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:09.057715    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hvw8v"
Jul 23 14:53:09.057862 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:53:09.057840    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hvw8v" podUID="413d2464-7258-4246-85bb-ac9f03976ee5"
Jul 23 14:53:10.057138 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:10.057111    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hvsnb"
Jul 23 14:53:10.057527 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:53:10.057263    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hvsnb" podUID="527775ad-7c13-4732-b824-4302ad79dcf4"
Jul 23 14:53:11.056803 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:11.056589    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hvw8v"
Jul 23 14:53:11.056970 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:11.056684    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7sqtc"
Jul 23 14:53:11.056970 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:53:11.056920    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hvw8v" podUID="413d2464-7258-4246-85bb-ac9f03976ee5"
Jul 23 14:53:11.057079 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:53:11.057029    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7sqtc" podUID="3370857a-3ffb-4603-bcad-503b48c10fd6"
Jul 23 14:53:12.057545 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:12.057516    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hvsnb"
Jul 23 14:53:12.057904 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:53:12.057620    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hvsnb" podUID="527775ad-7c13-4732-b824-4302ad79dcf4"
Jul 23 14:53:12.631948 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:12.631921    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/413d2464-7258-4246-85bb-ac9f03976ee5-metrics-certs\") pod \"network-metrics-daemon-hvw8v\" (UID: \"413d2464-7258-4246-85bb-ac9f03976ee5\") " pod="openshift-multus/network-metrics-daemon-hvw8v"
Jul 23 14:53:12.632082 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:53:12.632064    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 14:53:12.632140 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:53:12.632130    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/413d2464-7258-4246-85bb-ac9f03976ee5-metrics-certs podName:413d2464-7258-4246-85bb-ac9f03976ee5 nodeName:}" failed. No retries permitted until 2026-07-23 14:53:44.632110608 +0000 UTC m=+66.139984335 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/413d2464-7258-4246-85bb-ac9f03976ee5-metrics-certs") pod "network-metrics-daemon-hvw8v" (UID: "413d2464-7258-4246-85bb-ac9f03976ee5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 14:53:12.732819 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:12.732792    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/527775ad-7c13-4732-b824-4302ad79dcf4-original-pull-secret\") pod \"global-pull-secret-syncer-hvsnb\" (UID: \"527775ad-7c13-4732-b824-4302ad79dcf4\") " pod="kube-system/global-pull-secret-syncer-hvsnb"
Jul 23 14:53:12.732970 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:12.732823    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcqwm\" (UniqueName: \"kubernetes.io/projected/3370857a-3ffb-4603-bcad-503b48c10fd6-kube-api-access-fcqwm\") pod \"network-check-target-7sqtc\" (UID: \"3370857a-3ffb-4603-bcad-503b48c10fd6\") " pod="openshift-network-diagnostics/network-check-target-7sqtc"
Jul 23 14:53:12.732970 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:53:12.732925    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 23 14:53:12.732970 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:53:12.732935    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 23 14:53:12.732970 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:53:12.732951    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 23 14:53:12.732970 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:53:12.732963    2560 projected.go:196] Error preparing data for projected volume kube-api-access-fcqwm for pod openshift-network-diagnostics/network-check-target-7sqtc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 14:53:12.733122 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:53:12.732990    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/527775ad-7c13-4732-b824-4302ad79dcf4-original-pull-secret podName:527775ad-7c13-4732-b824-4302ad79dcf4 nodeName:}" failed. No retries permitted until 2026-07-23 14:53:44.732975788 +0000 UTC m=+66.240849514 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/527775ad-7c13-4732-b824-4302ad79dcf4-original-pull-secret") pod "global-pull-secret-syncer-hvsnb" (UID: "527775ad-7c13-4732-b824-4302ad79dcf4") : object "kube-system"/"original-pull-secret" not registered
Jul 23 14:53:12.733122 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:53:12.733020    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3370857a-3ffb-4603-bcad-503b48c10fd6-kube-api-access-fcqwm podName:3370857a-3ffb-4603-bcad-503b48c10fd6 nodeName:}" failed. No retries permitted until 2026-07-23 14:53:44.733009136 +0000 UTC m=+66.240882861 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-fcqwm" (UniqueName: "kubernetes.io/projected/3370857a-3ffb-4603-bcad-503b48c10fd6-kube-api-access-fcqwm") pod "network-check-target-7sqtc" (UID: "3370857a-3ffb-4603-bcad-503b48c10fd6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 14:53:13.057663 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:13.057587    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7sqtc"
Jul 23 14:53:13.058179 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:53:13.057710    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7sqtc" podUID="3370857a-3ffb-4603-bcad-503b48c10fd6"
Jul 23 14:53:13.058179 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:13.057758    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hvw8v"
Jul 23 14:53:13.058179 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:53:13.057844    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hvw8v" podUID="413d2464-7258-4246-85bb-ac9f03976ee5"
Jul 23 14:53:13.207274 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:13.207244    2560 generic.go:356] "Generic (PLEG): container finished" podID="6fbaa452-6ab5-4370-ae18-fb4290ce9c10" containerID="379d0e38abe81d6d942888353bcbaf3b2f82e3ed1a1c5ea2114e02a5e547dcd1" exitCode=0
Jul 23 14:53:13.207460 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:13.207280    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q79jt" event={"ID":"6fbaa452-6ab5-4370-ae18-fb4290ce9c10","Type":"ContainerDied","Data":"379d0e38abe81d6d942888353bcbaf3b2f82e3ed1a1c5ea2114e02a5e547dcd1"}
Jul 23 14:53:13.326585 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:13.326559    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-105.ec2.internal" event="NodeReady"
Jul 23 14:53:13.326727 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:13.326706    2560 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 23 14:53:13.381292 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:13.381259    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-scspg"]
Jul 23 14:53:13.389140 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:13.389116    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-sw5j8"]
Jul 23 14:53:13.389291 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:13.389247    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-scspg"
Jul 23 14:53:13.391934 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:13.391911    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 23 14:53:13.392045 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:13.392017    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-zzbs4\""
Jul 23 14:53:13.392082 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:13.392055    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 23 14:53:13.397810 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:13.397788    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-scspg"]
Jul 23 14:53:13.397928 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:13.397882    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-sw5j8"
Jul 23 14:53:13.400939 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:13.400916    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-sw5j8"]
Jul 23 14:53:13.401164 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:13.401114    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 23 14:53:13.401164 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:13.401161    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 23 14:53:13.401391 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:13.401218    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 23 14:53:13.401466 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:13.401432    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-hrh8d\""
Jul 23 14:53:13.537463 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:13.537423    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f7786ed8-2ad5-40ad-a866-ddacdead5ab1-cert\") pod \"ingress-canary-sw5j8\" (UID: \"f7786ed8-2ad5-40ad-a866-ddacdead5ab1\") " pod="openshift-ingress-canary/ingress-canary-sw5j8"
Jul 23 14:53:13.537463 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:13.537464    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzrkp\" (UniqueName: \"kubernetes.io/projected/a2a2b51d-3d22-42dd-96b6-093bdf7c3f7a-kube-api-access-xzrkp\") pod \"dns-default-scspg\" (UID: \"a2a2b51d-3d22-42dd-96b6-093bdf7c3f7a\") " pod="openshift-dns/dns-default-scspg"
Jul 23 14:53:13.537668 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:13.537488    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a2a2b51d-3d22-42dd-96b6-093bdf7c3f7a-metrics-tls\") pod \"dns-default-scspg\" (UID: \"a2a2b51d-3d22-42dd-96b6-093bdf7c3f7a\") " pod="openshift-dns/dns-default-scspg"
Jul 23 14:53:13.537668 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:13.537519    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4p4mg\" (UniqueName: \"kubernetes.io/projected/f7786ed8-2ad5-40ad-a866-ddacdead5ab1-kube-api-access-4p4mg\") pod \"ingress-canary-sw5j8\" (UID: \"f7786ed8-2ad5-40ad-a866-ddacdead5ab1\") " pod="openshift-ingress-canary/ingress-canary-sw5j8"
Jul 23 14:53:13.537668 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:13.537547    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2a2b51d-3d22-42dd-96b6-093bdf7c3f7a-tmp-dir\") pod \"dns-default-scspg\" (UID: \"a2a2b51d-3d22-42dd-96b6-093bdf7c3f7a\") " pod="openshift-dns/dns-default-scspg"
Jul 23 14:53:13.537668 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:13.537564    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a2a2b51d-3d22-42dd-96b6-093bdf7c3f7a-config-volume\") pod \"dns-default-scspg\" (UID: \"a2a2b51d-3d22-42dd-96b6-093bdf7c3f7a\") " pod="openshift-dns/dns-default-scspg"
Jul 23 14:53:13.638782 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:13.638695    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f7786ed8-2ad5-40ad-a866-ddacdead5ab1-cert\") pod \"ingress-canary-sw5j8\" (UID: \"f7786ed8-2ad5-40ad-a866-ddacdead5ab1\") " pod="openshift-ingress-canary/ingress-canary-sw5j8"
Jul 23 14:53:13.638782 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:13.638737    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzrkp\" (UniqueName: \"kubernetes.io/projected/a2a2b51d-3d22-42dd-96b6-093bdf7c3f7a-kube-api-access-xzrkp\") pod \"dns-default-scspg\" (UID: \"a2a2b51d-3d22-42dd-96b6-093bdf7c3f7a\") " pod="openshift-dns/dns-default-scspg"
Jul 23 14:53:13.638782 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:13.638757    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a2a2b51d-3d22-42dd-96b6-093bdf7c3f7a-metrics-tls\") pod \"dns-default-scspg\" (UID: \"a2a2b51d-3d22-42dd-96b6-093bdf7c3f7a\") " pod="openshift-dns/dns-default-scspg"
Jul 23 14:53:13.639190 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:53:13.638858    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 23 14:53:13.639190 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:13.638887    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4p4mg\" (UniqueName: \"kubernetes.io/projected/f7786ed8-2ad5-40ad-a866-ddacdead5ab1-kube-api-access-4p4mg\") pod \"ingress-canary-sw5j8\" (UID: \"f7786ed8-2ad5-40ad-a866-ddacdead5ab1\") " pod="openshift-ingress-canary/ingress-canary-sw5j8"
Jul 23 14:53:13.639190 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:53:13.638924    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2a2b51d-3d22-42dd-96b6-093bdf7c3f7a-metrics-tls podName:a2a2b51d-3d22-42dd-96b6-093bdf7c3f7a nodeName:}" failed. No retries permitted until 2026-07-23 14:53:14.138901471 +0000 UTC m=+35.646775205 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a2a2b51d-3d22-42dd-96b6-093bdf7c3f7a-metrics-tls") pod "dns-default-scspg" (UID: "a2a2b51d-3d22-42dd-96b6-093bdf7c3f7a") : secret "dns-default-metrics-tls" not found
Jul 23 14:53:13.639190 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:13.638965    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2a2b51d-3d22-42dd-96b6-093bdf7c3f7a-tmp-dir\") pod \"dns-default-scspg\" (UID: \"a2a2b51d-3d22-42dd-96b6-093bdf7c3f7a\") " pod="openshift-dns/dns-default-scspg"
Jul 23 14:53:13.639190 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:13.638993    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a2a2b51d-3d22-42dd-96b6-093bdf7c3f7a-config-volume\") pod \"dns-default-scspg\" (UID: \"a2a2b51d-3d22-42dd-96b6-093bdf7c3f7a\") " pod="openshift-dns/dns-default-scspg"
Jul 23 14:53:13.639464 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:13.639389    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2a2b51d-3d22-42dd-96b6-093bdf7c3f7a-tmp-dir\") pod \"dns-default-scspg\" (UID: \"a2a2b51d-3d22-42dd-96b6-093bdf7c3f7a\") " pod="openshift-dns/dns-default-scspg"
Jul 23 14:53:13.639581 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:13.639564    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a2a2b51d-3d22-42dd-96b6-093bdf7c3f7a-config-volume\") pod \"dns-default-scspg\" (UID: \"a2a2b51d-3d22-42dd-96b6-093bdf7c3f7a\") " pod="openshift-dns/dns-default-scspg"
Jul 23 14:53:13.643079 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:13.643053    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f7786ed8-2ad5-40ad-a866-ddacdead5ab1-cert\") pod \"ingress-canary-sw5j8\" (UID: \"f7786ed8-2ad5-40ad-a866-ddacdead5ab1\") " pod="openshift-ingress-canary/ingress-canary-sw5j8"
Jul 23 14:53:13.652938 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:13.652907    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4p4mg\" (UniqueName: \"kubernetes.io/projected/f7786ed8-2ad5-40ad-a866-ddacdead5ab1-kube-api-access-4p4mg\") pod \"ingress-canary-sw5j8\" (UID: \"f7786ed8-2ad5-40ad-a866-ddacdead5ab1\") " pod="openshift-ingress-canary/ingress-canary-sw5j8"
Jul 23 14:53:13.653050 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:13.652946    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzrkp\" (UniqueName: \"kubernetes.io/projected/a2a2b51d-3d22-42dd-96b6-093bdf7c3f7a-kube-api-access-xzrkp\") pod \"dns-default-scspg\" (UID: \"a2a2b51d-3d22-42dd-96b6-093bdf7c3f7a\") " pod="openshift-dns/dns-default-scspg"
Jul 23 14:53:13.713125 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:13.713087    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-sw5j8"
Jul 23 14:53:13.913021 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:13.912779    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-sw5j8"]
Jul 23 14:53:13.917871 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:53:13.917842    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7786ed8_2ad5_40ad_a866_ddacdead5ab1.slice/crio-eaf9e2f396ce6ea8b85fac163eb6708fe18ec2aab26335268c27465ec7f70403 WatchSource:0}: Error finding container eaf9e2f396ce6ea8b85fac163eb6708fe18ec2aab26335268c27465ec7f70403: Status 404 returned error can't find the container with id eaf9e2f396ce6ea8b85fac163eb6708fe18ec2aab26335268c27465ec7f70403
Jul 23 14:53:14.056906 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:14.056873    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hvsnb"
Jul 23 14:53:14.060447 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:14.060419    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 23 14:53:14.142798 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:14.142767    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a2a2b51d-3d22-42dd-96b6-093bdf7c3f7a-metrics-tls\") pod \"dns-default-scspg\" (UID: \"a2a2b51d-3d22-42dd-96b6-093bdf7c3f7a\") " pod="openshift-dns/dns-default-scspg"
Jul 23 14:53:14.142957 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:53:14.142910    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 23 14:53:14.142995 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:53:14.142968    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2a2b51d-3d22-42dd-96b6-093bdf7c3f7a-metrics-tls podName:a2a2b51d-3d22-42dd-96b6-093bdf7c3f7a nodeName:}" failed. No retries permitted until 2026-07-23 14:53:15.142952548 +0000 UTC m=+36.650826274 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a2a2b51d-3d22-42dd-96b6-093bdf7c3f7a-metrics-tls") pod "dns-default-scspg" (UID: "a2a2b51d-3d22-42dd-96b6-093bdf7c3f7a") : secret "dns-default-metrics-tls" not found
Jul 23 14:53:14.212312 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:14.212215    2560 generic.go:356] "Generic (PLEG): container finished" podID="6fbaa452-6ab5-4370-ae18-fb4290ce9c10" containerID="7a799c2503828f7120d09b920219826f6e4805a6a8359200b1bebbe2bded2c29" exitCode=0
Jul 23 14:53:14.212312 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:14.212263    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q79jt" event={"ID":"6fbaa452-6ab5-4370-ae18-fb4290ce9c10","Type":"ContainerDied","Data":"7a799c2503828f7120d09b920219826f6e4805a6a8359200b1bebbe2bded2c29"}
Jul 23 14:53:14.213476 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:14.213453    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-sw5j8" event={"ID":"f7786ed8-2ad5-40ad-a866-ddacdead5ab1","Type":"ContainerStarted","Data":"eaf9e2f396ce6ea8b85fac163eb6708fe18ec2aab26335268c27465ec7f70403"}
Jul 23 14:53:15.057461 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:15.057426    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hvw8v"
Jul 23 14:53:15.057766 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:15.057615    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7sqtc"
Jul 23 14:53:15.062271 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:15.062037    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 23 14:53:15.062271 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:15.062068    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-brw86\""
Jul 23 14:53:15.062271 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:15.062085    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 23 14:53:15.062271 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:15.062074    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 23 14:53:15.062271 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:15.062074    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-blqvw\""
Jul 23 14:53:15.151443 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:15.151406    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a2a2b51d-3d22-42dd-96b6-093bdf7c3f7a-metrics-tls\") pod \"dns-default-scspg\" (UID: \"a2a2b51d-3d22-42dd-96b6-093bdf7c3f7a\") " pod="openshift-dns/dns-default-scspg"
Jul 23 14:53:15.151609 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:53:15.151591    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 23 14:53:15.151673 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:53:15.151650    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2a2b51d-3d22-42dd-96b6-093bdf7c3f7a-metrics-tls podName:a2a2b51d-3d22-42dd-96b6-093bdf7c3f7a nodeName:}" failed. No retries permitted until 2026-07-23 14:53:17.151637504 +0000 UTC m=+38.659511230 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a2a2b51d-3d22-42dd-96b6-093bdf7c3f7a-metrics-tls") pod "dns-default-scspg" (UID: "a2a2b51d-3d22-42dd-96b6-093bdf7c3f7a") : secret "dns-default-metrics-tls" not found
Jul 23 14:53:15.220029 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:15.219993    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q79jt" event={"ID":"6fbaa452-6ab5-4370-ae18-fb4290ce9c10","Type":"ContainerStarted","Data":"4225e24dc1c104d448ac5f65de0c74bb6375e0d1acae32041171d88f72146714"}
Jul 23 14:53:15.245728 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:15.245672    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-q79jt" podStartSLOduration=7.344381531 podStartE2EDuration="36.245651551s" totalImagesPullingTime="28.90127002s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 14:52:39 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 14:53:15.243847502 +0000 UTC m=+36.751721250" watchObservedRunningTime="2026-07-23 14:53:15.245651551 +0000 UTC m=+36.753525303"
Jul 23 14:53:17.167424 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:17.167223    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a2a2b51d-3d22-42dd-96b6-093bdf7c3f7a-metrics-tls\") pod \"dns-default-scspg\" (UID: \"a2a2b51d-3d22-42dd-96b6-093bdf7c3f7a\") " pod="openshift-dns/dns-default-scspg"
Jul 23 14:53:17.167424 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:53:17.167372    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 23 14:53:17.167868 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:53:17.167491    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2a2b51d-3d22-42dd-96b6-093bdf7c3f7a-metrics-tls podName:a2a2b51d-3d22-42dd-96b6-093bdf7c3f7a nodeName:}" failed. No retries permitted until 2026-07-23 14:53:21.167468477 +0000 UTC m=+42.675342203 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a2a2b51d-3d22-42dd-96b6-093bdf7c3f7a-metrics-tls") pod "dns-default-scspg" (UID: "a2a2b51d-3d22-42dd-96b6-093bdf7c3f7a") : secret "dns-default-metrics-tls" not found
Jul 23 14:53:17.226767 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:17.226739    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-sw5j8" event={"ID":"f7786ed8-2ad5-40ad-a866-ddacdead5ab1","Type":"ContainerStarted","Data":"dc15530791f22ea2af8dea788e64c39aa7755ff0b598ce0ea2789b754a8e7d76"}
Jul 23 14:53:17.245179 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:17.245135    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-sw5j8" podStartSLOduration=1.781815147 podStartE2EDuration="4.245122785s" totalImagesPullingTime="2.463307638s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 14:53:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 14:53:17.243614695 +0000 UTC m=+38.751488445" watchObservedRunningTime="2026-07-23 14:53:17.245122785 +0000 UTC m=+38.752996532"
Jul 23 14:53:21.195873 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:21.195833    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a2a2b51d-3d22-42dd-96b6-093bdf7c3f7a-metrics-tls\") pod \"dns-default-scspg\" (UID: \"a2a2b51d-3d22-42dd-96b6-093bdf7c3f7a\") " pod="openshift-dns/dns-default-scspg"
Jul 23 14:53:21.196376 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:53:21.195960    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 23 14:53:21.196376 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:53:21.196011    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2a2b51d-3d22-42dd-96b6-093bdf7c3f7a-metrics-tls podName:a2a2b51d-3d22-42dd-96b6-093bdf7c3f7a nodeName:}" failed. No retries permitted until 2026-07-23 14:53:29.195998265 +0000 UTC m=+50.703871991 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a2a2b51d-3d22-42dd-96b6-093bdf7c3f7a-metrics-tls") pod "dns-default-scspg" (UID: "a2a2b51d-3d22-42dd-96b6-093bdf7c3f7a") : secret "dns-default-metrics-tls" not found
Jul 23 14:53:29.250889 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:29.250847    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a2a2b51d-3d22-42dd-96b6-093bdf7c3f7a-metrics-tls\") pod \"dns-default-scspg\" (UID: \"a2a2b51d-3d22-42dd-96b6-093bdf7c3f7a\") " pod="openshift-dns/dns-default-scspg"
Jul 23 14:53:29.251328 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:53:29.250995    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 23 14:53:29.251328 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:53:29.251061    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2a2b51d-3d22-42dd-96b6-093bdf7c3f7a-metrics-tls podName:a2a2b51d-3d22-42dd-96b6-093bdf7c3f7a nodeName:}" failed. No retries permitted until 2026-07-23 14:53:45.251045583 +0000 UTC m=+66.758919309 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a2a2b51d-3d22-42dd-96b6-093bdf7c3f7a-metrics-tls") pod "dns-default-scspg" (UID: "a2a2b51d-3d22-42dd-96b6-093bdf7c3f7a") : secret "dns-default-metrics-tls" not found
Jul 23 14:53:39.205703 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:39.205672    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-z4fvn"
Jul 23 14:53:44.658355 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:44.658291    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/413d2464-7258-4246-85bb-ac9f03976ee5-metrics-certs\") pod \"network-metrics-daemon-hvw8v\" (UID: \"413d2464-7258-4246-85bb-ac9f03976ee5\") " pod="openshift-multus/network-metrics-daemon-hvw8v"
Jul 23 14:53:44.661354 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:44.661333    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 23 14:53:44.668853 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:53:44.668830    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 23 14:53:44.668917 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:53:44.668907    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/413d2464-7258-4246-85bb-ac9f03976ee5-metrics-certs podName:413d2464-7258-4246-85bb-ac9f03976ee5 nodeName:}" failed. No retries permitted until 2026-07-23 14:54:48.668890059 +0000 UTC m=+130.176763785 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/413d2464-7258-4246-85bb-ac9f03976ee5-metrics-certs") pod "network-metrics-daemon-hvw8v" (UID: "413d2464-7258-4246-85bb-ac9f03976ee5") : secret "metrics-daemon-secret" not found
Jul 23 14:53:44.759514 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:44.759470    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/527775ad-7c13-4732-b824-4302ad79dcf4-original-pull-secret\") pod \"global-pull-secret-syncer-hvsnb\" (UID: \"527775ad-7c13-4732-b824-4302ad79dcf4\") " pod="kube-system/global-pull-secret-syncer-hvsnb"
Jul 23 14:53:44.759514 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:44.759517    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcqwm\" (UniqueName: \"kubernetes.io/projected/3370857a-3ffb-4603-bcad-503b48c10fd6-kube-api-access-fcqwm\") pod \"network-check-target-7sqtc\" (UID: \"3370857a-3ffb-4603-bcad-503b48c10fd6\") " pod="openshift-network-diagnostics/network-check-target-7sqtc"
Jul 23 14:53:44.762575 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:44.762557    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 23 14:53:44.762619 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:44.762577    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 23 14:53:44.772697 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:44.772674    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 23 14:53:44.773857 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:44.773836    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/527775ad-7c13-4732-b824-4302ad79dcf4-original-pull-secret\") pod \"global-pull-secret-syncer-hvsnb\" (UID: \"527775ad-7c13-4732-b824-4302ad79dcf4\") " pod="kube-system/global-pull-secret-syncer-hvsnb"
Jul 23 14:53:44.782339 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:44.782314    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcqwm\" (UniqueName: \"kubernetes.io/projected/3370857a-3ffb-4603-bcad-503b48c10fd6-kube-api-access-fcqwm\") pod \"network-check-target-7sqtc\" (UID: \"3370857a-3ffb-4603-bcad-503b48c10fd6\") " pod="openshift-network-diagnostics/network-check-target-7sqtc"
Jul 23 14:53:44.966025 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:44.965997    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hvsnb"
Jul 23 14:53:45.078013 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:45.077980    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-blqvw\""
Jul 23 14:53:45.085267 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:45.085243    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7sqtc"
Jul 23 14:53:45.090200 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:45.090177    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-hvsnb"]
Jul 23 14:53:45.094124 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:53:45.094099    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod527775ad_7c13_4732_b824_4302ad79dcf4.slice/crio-d0d263b1183e636a6fc4ff3f2794df9121788ee160b414d2ddf6cabfc3507bac WatchSource:0}: Error finding container d0d263b1183e636a6fc4ff3f2794df9121788ee160b414d2ddf6cabfc3507bac: Status 404 returned error can't find the container with id d0d263b1183e636a6fc4ff3f2794df9121788ee160b414d2ddf6cabfc3507bac
Jul 23 14:53:45.208300 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:45.208250    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-7sqtc"]
Jul 23 14:53:45.211656 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:53:45.211630    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3370857a_3ffb_4603_bcad_503b48c10fd6.slice/crio-bcb1cb2a5ed27fc6d9bdfa619d00d46502b45a9d2bb82725a67def306ea78a8e WatchSource:0}: Error finding container bcb1cb2a5ed27fc6d9bdfa619d00d46502b45a9d2bb82725a67def306ea78a8e: Status 404 returned error can't find the container with id bcb1cb2a5ed27fc6d9bdfa619d00d46502b45a9d2bb82725a67def306ea78a8e
Jul 23 14:53:45.263981 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:45.263894    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a2a2b51d-3d22-42dd-96b6-093bdf7c3f7a-metrics-tls\") pod \"dns-default-scspg\" (UID: \"a2a2b51d-3d22-42dd-96b6-093bdf7c3f7a\") " pod="openshift-dns/dns-default-scspg"
Jul 23 14:53:45.264116 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:53:45.264011    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 23 14:53:45.264116 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:53:45.264076    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2a2b51d-3d22-42dd-96b6-093bdf7c3f7a-metrics-tls podName:a2a2b51d-3d22-42dd-96b6-093bdf7c3f7a nodeName:}" failed. No retries permitted until 2026-07-23 14:54:17.264062331 +0000 UTC m=+98.771936057 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a2a2b51d-3d22-42dd-96b6-093bdf7c3f7a-metrics-tls") pod "dns-default-scspg" (UID: "a2a2b51d-3d22-42dd-96b6-093bdf7c3f7a") : secret "dns-default-metrics-tls" not found
Jul 23 14:53:45.293247 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:45.293200    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-7sqtc" event={"ID":"3370857a-3ffb-4603-bcad-503b48c10fd6","Type":"ContainerStarted","Data":"bcb1cb2a5ed27fc6d9bdfa619d00d46502b45a9d2bb82725a67def306ea78a8e"}
Jul 23 14:53:45.296223 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:45.296191    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-hvsnb" event={"ID":"527775ad-7c13-4732-b824-4302ad79dcf4","Type":"ContainerStarted","Data":"d0d263b1183e636a6fc4ff3f2794df9121788ee160b414d2ddf6cabfc3507bac"}
Jul 23 14:53:50.311652 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:50.311609    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-7sqtc" event={"ID":"3370857a-3ffb-4603-bcad-503b48c10fd6","Type":"ContainerStarted","Data":"f01ff9d9d894ba7d1582b898a5cf81bd920ef728e6d7b133bfbaff2f58b56b94"}
Jul 23 14:53:50.312210 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:50.311733    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-7sqtc"
Jul 23 14:53:50.313047 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:50.313025    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-hvsnb" event={"ID":"527775ad-7c13-4732-b824-4302ad79dcf4","Type":"ContainerStarted","Data":"1c0efb2658f7f0f4e8300a04e811907a6e0c7b15563f19453f6b90ea8d8e5a24"}
Jul 23 14:53:50.331114 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:50.331065    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-7sqtc" podStartSLOduration=67.069913603 podStartE2EDuration="1m11.331048129s" totalImagesPullingTime="4.261134526s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 14:52:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 14:53:50.330815344 +0000 UTC m=+71.838689091" watchObservedRunningTime="2026-07-23 14:53:50.331048129 +0000 UTC m=+71.838921890"
Jul 23 14:53:50.347278 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:50.347209    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-hvsnb" podStartSLOduration=66.964249202 podStartE2EDuration="1m11.347195504s" totalImagesPullingTime="4.382946302s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 14:52:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 14:53:50.346976184 +0000 UTC m=+71.854849932" watchObservedRunningTime="2026-07-23 14:53:50.347195504 +0000 UTC m=+71.855069251"
Jul 23 14:53:59.514588 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:59.514475    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c659bb677-6pnk8"]
Jul 23 14:53:59.516365 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:59.516343    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c659bb677-6pnk8"
Jul 23 14:53:59.521056 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:59.521034    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 23 14:53:59.522571 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:59.522539    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 23 14:53:59.522571 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:59.522540    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 23 14:53:59.522743 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:59.522669    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 23 14:53:59.528744 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:59.528718    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c659bb677-6pnk8"]
Jul 23 14:53:59.561089 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:59.561059    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vhpq\" (UniqueName: \"kubernetes.io/projected/39effd7f-8195-471d-af4f-d9b9c780e4a4-kube-api-access-2vhpq\") pod \"klusterlet-addon-workmgr-5c659bb677-6pnk8\" (UID: \"39effd7f-8195-471d-af4f-d9b9c780e4a4\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c659bb677-6pnk8"
Jul 23 14:53:59.561249 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:59.561098    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/39effd7f-8195-471d-af4f-d9b9c780e4a4-tmp\") pod \"klusterlet-addon-workmgr-5c659bb677-6pnk8\" (UID: \"39effd7f-8195-471d-af4f-d9b9c780e4a4\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c659bb677-6pnk8"
Jul 23 14:53:59.561249 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:59.561158    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/39effd7f-8195-471d-af4f-d9b9c780e4a4-klusterlet-config\") pod \"klusterlet-addon-workmgr-5c659bb677-6pnk8\" (UID: \"39effd7f-8195-471d-af4f-d9b9c780e4a4\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c659bb677-6pnk8"
Jul 23 14:53:59.563630 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:59.563610    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-85986d785b-kjz9f"]
Jul 23 14:53:59.565679 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:59.565665    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-85986d785b-kjz9f"
Jul 23 14:53:59.568329 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:59.568306    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Jul 23 14:53:59.568420 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:59.568349    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-ws6g7\""
Jul 23 14:53:59.577867 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:59.577847    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-85986d785b-kjz9f"]
Jul 23 14:53:59.661703 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:59.661672    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/39effd7f-8195-471d-af4f-d9b9c780e4a4-klusterlet-config\") pod \"klusterlet-addon-workmgr-5c659bb677-6pnk8\" (UID: \"39effd7f-8195-471d-af4f-d9b9c780e4a4\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c659bb677-6pnk8"
Jul 23 14:53:59.661703 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:59.661708    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4f650a95-e179-4c8f-8e34-916ee32504e3-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-85986d785b-kjz9f\" (UID: \"4f650a95-e179-4c8f-8e34-916ee32504e3\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-85986d785b-kjz9f"
Jul 23 14:53:59.661900 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:59.661741    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vhpq\" (UniqueName: \"kubernetes.io/projected/39effd7f-8195-471d-af4f-d9b9c780e4a4-kube-api-access-2vhpq\") pod \"klusterlet-addon-workmgr-5c659bb677-6pnk8\" (UID: \"39effd7f-8195-471d-af4f-d9b9c780e4a4\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c659bb677-6pnk8"
Jul 23 14:53:59.661900 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:59.661786    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvqt8\" (UniqueName: \"kubernetes.io/projected/4f650a95-e179-4c8f-8e34-916ee32504e3-kube-api-access-vvqt8\") pod \"managed-serviceaccount-addon-agent-85986d785b-kjz9f\" (UID: \"4f650a95-e179-4c8f-8e34-916ee32504e3\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-85986d785b-kjz9f"
Jul 23 14:53:59.661900 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:59.661839    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/39effd7f-8195-471d-af4f-d9b9c780e4a4-tmp\") pod \"klusterlet-addon-workmgr-5c659bb677-6pnk8\" (UID: \"39effd7f-8195-471d-af4f-d9b9c780e4a4\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c659bb677-6pnk8"
Jul 23 14:53:59.662159 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:59.662134    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/39effd7f-8195-471d-af4f-d9b9c780e4a4-tmp\") pod \"klusterlet-addon-workmgr-5c659bb677-6pnk8\" (UID: \"39effd7f-8195-471d-af4f-d9b9c780e4a4\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c659bb677-6pnk8"
Jul 23 14:53:59.664206 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:59.664190    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/39effd7f-8195-471d-af4f-d9b9c780e4a4-klusterlet-config\") pod \"klusterlet-addon-workmgr-5c659bb677-6pnk8\" (UID: \"39effd7f-8195-471d-af4f-d9b9c780e4a4\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c659bb677-6pnk8"
Jul 23 14:53:59.672987 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:59.672963    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vhpq\" (UniqueName: \"kubernetes.io/projected/39effd7f-8195-471d-af4f-d9b9c780e4a4-kube-api-access-2vhpq\") pod \"klusterlet-addon-workmgr-5c659bb677-6pnk8\" (UID: \"39effd7f-8195-471d-af4f-d9b9c780e4a4\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c659bb677-6pnk8"
Jul 23 14:53:59.766160 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:59.764142    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4f650a95-e179-4c8f-8e34-916ee32504e3-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-85986d785b-kjz9f\" (UID: \"4f650a95-e179-4c8f-8e34-916ee32504e3\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-85986d785b-kjz9f"
Jul 23 14:53:59.766160 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:59.764256    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvqt8\" (UniqueName: \"kubernetes.io/projected/4f650a95-e179-4c8f-8e34-916ee32504e3-kube-api-access-vvqt8\") pod \"managed-serviceaccount-addon-agent-85986d785b-kjz9f\" (UID: \"4f650a95-e179-4c8f-8e34-916ee32504e3\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-85986d785b-kjz9f"
Jul 23 14:53:59.767502 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:59.767469    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4f650a95-e179-4c8f-8e34-916ee32504e3-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-85986d785b-kjz9f\" (UID: \"4f650a95-e179-4c8f-8e34-916ee32504e3\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-85986d785b-kjz9f"
Jul 23 14:53:59.777903 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:59.777881    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvqt8\" (UniqueName: \"kubernetes.io/projected/4f650a95-e179-4c8f-8e34-916ee32504e3-kube-api-access-vvqt8\") pod \"managed-serviceaccount-addon-agent-85986d785b-kjz9f\" (UID: \"4f650a95-e179-4c8f-8e34-916ee32504e3\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-85986d785b-kjz9f"
Jul 23 14:53:59.824952 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:59.824923    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c659bb677-6pnk8"
Jul 23 14:53:59.885358 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:59.885324    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-85986d785b-kjz9f"
Jul 23 14:53:59.948469 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:53:59.948441    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c659bb677-6pnk8"]
Jul 23 14:53:59.951792 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:53:59.951764    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39effd7f_8195_471d_af4f_d9b9c780e4a4.slice/crio-810f8c5a1de22a623b92b5c708ba13b1eef7650aaa9106b6a95bc7e64c6dcb72 WatchSource:0}: Error finding container 810f8c5a1de22a623b92b5c708ba13b1eef7650aaa9106b6a95bc7e64c6dcb72: Status 404 returned error can't find the container with id 810f8c5a1de22a623b92b5c708ba13b1eef7650aaa9106b6a95bc7e64c6dcb72
Jul 23 14:54:00.009649 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:54:00.009615    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-85986d785b-kjz9f"]
Jul 23 14:54:00.013088 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:54:00.013052    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4f650a95_e179_4c8f_8e34_916ee32504e3.slice/crio-95990395c7524aa68be0c5627450292467fcc566a976b927e27b6def55e8282a WatchSource:0}: Error finding container 95990395c7524aa68be0c5627450292467fcc566a976b927e27b6def55e8282a: Status 404 returned error can't find the container with id 95990395c7524aa68be0c5627450292467fcc566a976b927e27b6def55e8282a
Jul 23 14:54:00.337002 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:54:00.336905    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c659bb677-6pnk8" event={"ID":"39effd7f-8195-471d-af4f-d9b9c780e4a4","Type":"ContainerStarted","Data":"810f8c5a1de22a623b92b5c708ba13b1eef7650aaa9106b6a95bc7e64c6dcb72"}
Jul 23 14:54:00.337969 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:54:00.337939    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-85986d785b-kjz9f" event={"ID":"4f650a95-e179-4c8f-8e34-916ee32504e3","Type":"ContainerStarted","Data":"95990395c7524aa68be0c5627450292467fcc566a976b927e27b6def55e8282a"}
Jul 23 14:54:04.351048 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:54:04.350957    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c659bb677-6pnk8" event={"ID":"39effd7f-8195-471d-af4f-d9b9c780e4a4","Type":"ContainerStarted","Data":"929f5768fac95c622dd54d0c56e5d0348e6f2be5db6c2bbc1a6b7bb889a8dd1d"}
Jul 23 14:54:04.351048 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:54:04.351023    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c659bb677-6pnk8"
Jul 23 14:54:04.352430 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:54:04.352395    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-85986d785b-kjz9f" event={"ID":"4f650a95-e179-4c8f-8e34-916ee32504e3","Type":"ContainerStarted","Data":"ed47e0f9c6ce36907616a023f516c82e1d4b12df55587ae29c2ed24b98207607"}
Jul 23 14:54:04.352823 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:54:04.352809    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c659bb677-6pnk8"
Jul 23 14:54:04.424387 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:54:04.424343    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c659bb677-6pnk8" podStartSLOduration=1.282181008 podStartE2EDuration="5.424329546s" totalImagesPullingTime="4.142148538s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 14:53:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 14:54:04.388376186 +0000 UTC m=+85.896249946" watchObservedRunningTime="2026-07-23 14:54:04.424329546 +0000 UTC m=+85.932203294"
Jul 23 14:54:04.424725 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:54:04.424704    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-85986d785b-kjz9f" podStartSLOduration=1.35649916 podStartE2EDuration="5.424698898s" totalImagesPullingTime="4.068199738s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 14:53:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 14:54:04.424421901 +0000 UTC m=+85.932295653" watchObservedRunningTime="2026-07-23 14:54:04.424698898 +0000 UTC m=+85.932572646"
Jul 23 14:54:17.290398 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:54:17.290343    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a2a2b51d-3d22-42dd-96b6-093bdf7c3f7a-metrics-tls\") pod \"dns-default-scspg\" (UID: \"a2a2b51d-3d22-42dd-96b6-093bdf7c3f7a\") " pod="openshift-dns/dns-default-scspg"
Jul 23 14:54:17.290791 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:54:17.290494    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 23 14:54:17.290791 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:54:17.290575    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2a2b51d-3d22-42dd-96b6-093bdf7c3f7a-metrics-tls podName:a2a2b51d-3d22-42dd-96b6-093bdf7c3f7a nodeName:}" failed. No retries permitted until 2026-07-23 14:55:21.290558067 +0000 UTC m=+162.798431793 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a2a2b51d-3d22-42dd-96b6-093bdf7c3f7a-metrics-tls") pod "dns-default-scspg" (UID: "a2a2b51d-3d22-42dd-96b6-093bdf7c3f7a") : secret "dns-default-metrics-tls" not found
Jul 23 14:54:21.318091 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:54:21.318064    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-7sqtc"
Jul 23 14:54:48.711810 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:54:48.711771    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/413d2464-7258-4246-85bb-ac9f03976ee5-metrics-certs\") pod \"network-metrics-daemon-hvw8v\" (UID: \"413d2464-7258-4246-85bb-ac9f03976ee5\") " pod="openshift-multus/network-metrics-daemon-hvw8v"
Jul 23 14:54:48.712306 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:54:48.711929    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 23 14:54:48.712306 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:54:48.712000    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/413d2464-7258-4246-85bb-ac9f03976ee5-metrics-certs podName:413d2464-7258-4246-85bb-ac9f03976ee5 nodeName:}" failed. No retries permitted until 2026-07-23 14:56:50.711981663 +0000 UTC m=+252.219855390 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/413d2464-7258-4246-85bb-ac9f03976ee5-metrics-certs") pod "network-metrics-daemon-hvw8v" (UID: "413d2464-7258-4246-85bb-ac9f03976ee5") : secret "metrics-daemon-secret" not found
Jul 23 14:55:16.408382 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:55:16.408336    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-scspg" podUID="a2a2b51d-3d22-42dd-96b6-093bdf7c3f7a"
Jul 23 14:55:16.552249 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:55:16.552204    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-scspg"
Jul 23 14:55:18.068830 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:55:18.068784    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-hvw8v" podUID="413d2464-7258-4246-85bb-ac9f03976ee5"
Jul 23 14:55:21.336642 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:55:21.336602    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a2a2b51d-3d22-42dd-96b6-093bdf7c3f7a-metrics-tls\") pod \"dns-default-scspg\" (UID: \"a2a2b51d-3d22-42dd-96b6-093bdf7c3f7a\") " pod="openshift-dns/dns-default-scspg"
Jul 23 14:55:21.337017 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:55:21.336743    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 23 14:55:21.337017 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:55:21.336829    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2a2b51d-3d22-42dd-96b6-093bdf7c3f7a-metrics-tls podName:a2a2b51d-3d22-42dd-96b6-093bdf7c3f7a nodeName:}" failed. No retries permitted until 2026-07-23 14:57:23.33681291 +0000 UTC m=+284.844686636 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a2a2b51d-3d22-42dd-96b6-093bdf7c3f7a-metrics-tls") pod "dns-default-scspg" (UID: "a2a2b51d-3d22-42dd-96b6-093bdf7c3f7a") : secret "dns-default-metrics-tls" not found
Jul 23 14:55:29.058182 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:55:29.058142    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hvw8v"
Jul 23 14:56:04.351663 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:56:04.351607    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c659bb677-6pnk8" podUID="39effd7f-8195-471d-af4f-d9b9c780e4a4" containerName="acm-agent" probeResult="failure" output="Get \"http://10.134.0.8:8000/readyz\": dial tcp 10.134.0.8:8000: connect: connection refused"
Jul 23 14:56:04.690280 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:56:04.690246    2560 generic.go:356] "Generic (PLEG): container finished" podID="39effd7f-8195-471d-af4f-d9b9c780e4a4" containerID="929f5768fac95c622dd54d0c56e5d0348e6f2be5db6c2bbc1a6b7bb889a8dd1d" exitCode=1
Jul 23 14:56:04.690461 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:56:04.690305    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c659bb677-6pnk8" event={"ID":"39effd7f-8195-471d-af4f-d9b9c780e4a4","Type":"ContainerDied","Data":"929f5768fac95c622dd54d0c56e5d0348e6f2be5db6c2bbc1a6b7bb889a8dd1d"}
Jul 23 14:56:04.690633 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:56:04.690614    2560 scope.go:117] "RemoveContainer" containerID="929f5768fac95c622dd54d0c56e5d0348e6f2be5db6c2bbc1a6b7bb889a8dd1d"
Jul 23 14:56:04.691894 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:56:04.691851    2560 generic.go:356] "Generic (PLEG): container finished" podID="4f650a95-e179-4c8f-8e34-916ee32504e3" containerID="ed47e0f9c6ce36907616a023f516c82e1d4b12df55587ae29c2ed24b98207607" exitCode=255
Jul 23 14:56:04.691894 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:56:04.691884    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-85986d785b-kjz9f" event={"ID":"4f650a95-e179-4c8f-8e34-916ee32504e3","Type":"ContainerDied","Data":"ed47e0f9c6ce36907616a023f516c82e1d4b12df55587ae29c2ed24b98207607"}
Jul 23 14:56:04.697697 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:56:04.697679    2560 scope.go:117] "RemoveContainer" containerID="ed47e0f9c6ce36907616a023f516c82e1d4b12df55587ae29c2ed24b98207607"
Jul 23 14:56:05.696701 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:56:05.696662    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c659bb677-6pnk8" event={"ID":"39effd7f-8195-471d-af4f-d9b9c780e4a4","Type":"ContainerStarted","Data":"90c755368f794b7ee44702c6ff4b1d4e2e31a1fab668a4e2e0892be0cd075ad3"}
Jul 23 14:56:05.697140 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:56:05.696985    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c659bb677-6pnk8"
Jul 23 14:56:05.697616 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:56:05.697600    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c659bb677-6pnk8"
Jul 23 14:56:05.698565 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:56:05.698542    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-85986d785b-kjz9f" event={"ID":"4f650a95-e179-4c8f-8e34-916ee32504e3","Type":"ContainerStarted","Data":"5e993bcd7e4a62cf6e9eeeb805ff3a90b28b9a7185f9994c1b0d5ad7d0fb3cff"}
Jul 23 14:56:44.220881 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:56:44.220845    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-kv62p"]
Jul 23 14:56:44.223810 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:56:44.223792    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-kv62p"
Jul 23 14:56:44.226671 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:56:44.226645    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Jul 23 14:56:44.226773 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:56:44.226685    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Jul 23 14:56:44.226773 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:56:44.226685    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Jul 23 14:56:44.228009 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:56:44.227995    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-p5hdt\""
Jul 23 14:56:44.236140 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:56:44.236118    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-kv62p"]
Jul 23 14:56:44.292965 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:56:44.292932    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrnr8\" (UniqueName: \"kubernetes.io/projected/357d9f32-903d-472b-bf56-f5151ba1ca8b-kube-api-access-xrnr8\") pod \"cluster-samples-operator-79b7c55f87-kv62p\" (UID: \"357d9f32-903d-472b-bf56-f5151ba1ca8b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-kv62p"
Jul 23 14:56:44.293107 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:56:44.292995    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/357d9f32-903d-472b-bf56-f5151ba1ca8b-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-kv62p\" (UID: \"357d9f32-903d-472b-bf56-f5151ba1ca8b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-kv62p"
Jul 23 14:56:44.313515 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:56:44.313490    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-k5zd5"]
Jul 23 14:56:44.317272 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:56:44.317250    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-555d9d7885-zndrp"]
Jul 23 14:56:44.317367 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:56:44.317354    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-k5zd5"
Jul 23 14:56:44.319344 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:56:44.319326    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5cf48ff8c9-p9xlr"]
Jul 23 14:56:44.319453 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:56:44.319437    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-555d9d7885-zndrp"
Jul 23 14:56:44.322063 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:56:44.322045    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5cf48ff8c9-p9xlr"
Jul 23 14:56:44.322552 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:56:44.322527    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 23 14:56:44.323001 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:56:44.322983    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Jul 23 14:56:44.323049 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:56:44.323016    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Jul 23 14:56:44.323366 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:56:44.323351    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Jul 23 14:56:44.323629 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:56:44.323614    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-p76qm\""
Jul 23 14:56:44.324562 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:56:44.324541    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Jul 23 14:56:44.324865 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:56:44.324837    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-f7d2x\""
Jul 23 14:56:44.325069 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:56:44.325044    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 23 14:56:44.325960 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:56:44.325878    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 23 14:56:44.325960 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:56:44.325891    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-9zwgm\""
Jul 23 14:56:44.326123 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:56:44.325981    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Jul 23 14:56:44.326123 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:56:44.326036    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 23 14:56:44.326318 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:56:44.326300    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 23 14:56:44.326439 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:56:44.326340    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Jul 23 14:56:44.330636 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:56:44.330593    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Jul 23 14:56:44.333366 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:56:44.333345    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-555d9d7885-zndrp"]
Jul 23 14:56:44.333907 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:56:44.333754    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 23 14:56:44.334264 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:56:44.334219    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-k5zd5"]
Jul 23 14:56:44.339706 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:56:44.339681    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5cf48ff8c9-p9xlr"]
Jul 23 14:56:44.393665 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:56:44.393636    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a253789c-c1c0-409f-b335-63fe5f2fb4af-serving-cert\") pod \"insights-operator-555d9d7885-zndrp\" (UID: \"a253789c-c1c0-409f-b335-63fe5f2fb4af\") " pod="openshift-insights/insights-operator-555d9d7885-zndrp"
Jul 23 14:56:44.393665 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:56:44.393668    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7df58b86-f921-4316-b8b1-f48137124a76-image-registry-private-configuration\") pod \"image-registry-5cf48ff8c9-p9xlr\" (UID: \"7df58b86-f921-4316-b8b1-f48137124a76\") " pod="openshift-image-registry/image-registry-5cf48ff8c9-p9xlr"
Jul 23 14:56:44.393851 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:56:44.393686    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7df58b86-f921-4316-b8b1-f48137124a76-bound-sa-token\") pod \"image-registry-5cf48ff8c9-p9xlr\" (UID: \"7df58b86-f921-4316-b8b1-f48137124a76\") " pod="openshift-image-registry/image-registry-5cf48ff8c9-p9xlr"
Jul 23 14:56:44.393851 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:56:44.393706    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7df58b86-f921-4316-b8b1-f48137124a76-trusted-ca\") pod \"image-registry-5cf48ff8c9-p9xlr\" (UID: \"7df58b86-f921-4316-b8b1-f48137124a76\") " pod="openshift-image-registry/image-registry-5cf48ff8c9-p9xlr"
Jul 23 14:56:44.393851 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:56:44.393724    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9ae597eb-68e0-447c-bf0c-781a80684af1-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-k5zd5\" (UID: \"9ae597eb-68e0-447c-bf0c-781a80684af1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-k5zd5"
Jul 23 14:56:44.393851 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:56:44.393777    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7df58b86-f921-4316-b8b1-f48137124a76-installation-pull-secrets\") pod \"image-registry-5cf48ff8c9-p9xlr\" (UID: \"7df58b86-f921-4316-b8b1-f48137124a76\") " pod="openshift-image-registry/image-registry-5cf48ff8c9-p9xlr"
Jul 23 14:56:44.393851 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:56:44.393808    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fjj5\" (UniqueName: \"kubernetes.io/projected/7df58b86-f921-4316-b8b1-f48137124a76-kube-api-access-9fjj5\") pod \"image-registry-5cf48ff8c9-p9xlr\" (UID: \"7df58b86-f921-4316-b8b1-f48137124a76\") " pod="openshift-image-registry/image-registry-5cf48ff8c9-p9xlr"
Jul 23 14:56:44.393851 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:56:44.393851    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7df58b86-f921-4316-b8b1-f48137124a76-registry-tls\") pod \"image-registry-5cf48ff8c9-p9xlr\" (UID: \"7df58b86-f921-4316-b8b1-f48137124a76\") " pod="openshift-image-registry/image-registry-5cf48ff8c9-p9xlr"
Jul 23 14:56:44.394111 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:56:44.393868    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7df58b86-f921-4316-b8b1-f48137124a76-registry-certificates\") pod \"image-registry-5cf48ff8c9-p9xlr\" (UID: \"7df58b86-f921-4316-b8b1-f48137124a76\") " pod="openshift-image-registry/image-registry-5cf48ff8c9-p9xlr"
Jul 23 14:56:44.394111 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:56:44.393912    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrwcp\" (UniqueName: \"kubernetes.io/projected/9ae597eb-68e0-447c-bf0c-781a80684af1-kube-api-access-jrwcp\") pod \"kube-storage-version-migrator-operator-7658d9b749-k5zd5\" (UID: \"9ae597eb-68e0-447c-bf0c-781a80684af1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-k5zd5"
Jul 23 14:56:44.394111 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:56:44.393931    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9ae597eb-68e0-447c-bf0c-781a80684af1-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-k5zd5\" (UID: \"9ae597eb-68e0-447c-bf0c-781a80684af1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-k5zd5"
Jul 23 14:56:44.394111 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:56:44.393948    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7df58b86-f921-4316-b8b1-f48137124a76-ca-trust-extracted\") pod \"image-registry-5cf48ff8c9-p9xlr\" (UID: \"7df58b86-f921-4316-b8b1-f48137124a76\") " pod="openshift-image-registry/image-registry-5cf48ff8c9-p9xlr"
Jul 23 14:56:44.394111 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:56:44.393964    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a253789c-c1c0-409f-b335-63fe5f2fb4af-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-zndrp\" (UID: \"a253789c-c1c0-409f-b335-63fe5f2fb4af\") " pod="openshift-insights/insights-operator-555d9d7885-zndrp"
Jul 23 14:56:44.394111 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:56:44.394057    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a253789c-c1c0-409f-b335-63fe5f2fb4af-service-ca-bundle\") pod \"insights-operator-555d9d7885-zndrp\" (UID: \"a253789c-c1c0-409f-b335-63fe5f2fb4af\") " pod="openshift-insights/insights-operator-555d9d7885-zndrp"
Jul 23 14:56:44.394111 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:56:44.394095    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/357d9f32-903d-472b-bf56-f5151ba1ca8b-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-kv62p\" (UID: \"357d9f32-903d-472b-bf56-f5151ba1ca8b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-kv62p"
Jul 23 14:56:44.394462 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:56:44.394126    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/a253789c-c1c0-409f-b335-63fe5f2fb4af-snapshots\") pod \"insights-operator-555d9d7885-zndrp\" (UID: \"a253789c-c1c0-409f-b335-63fe5f2fb4af\") " pod="openshift-insights/insights-operator-555d9d7885-zndrp"
Jul 23 14:56:44.394462 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:56:44.394213    2560 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 23 14:56:44.394462 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:56:44.394259    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7jdh\" (UniqueName: \"kubernetes.io/projected/a253789c-c1c0-409f-b335-63fe5f2fb4af-kube-api-access-w7jdh\") pod \"insights-operator-555d9d7885-zndrp\" (UID: \"a253789c-c1c0-409f-b335-63fe5f2fb4af\") " pod="openshift-insights/insights-operator-555d9d7885-zndrp"
Jul 23 14:56:44.394462 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:56:44.394276    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/357d9f32-903d-472b-bf56-f5151ba1ca8b-samples-operator-tls podName:357d9f32-903d-472b-bf56-f5151ba1ca8b nodeName:}" failed. No retries permitted until 2026-07-23 14:56:44.894262431 +0000 UTC m=+246.402136157 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/357d9f32-903d-472b-bf56-f5151ba1ca8b-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-kv62p" (UID: "357d9f32-903d-472b-bf56-f5151ba1ca8b") : secret "samples-operator-tls" not found
Jul 23 14:56:44.394462 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:56:44.394316    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrnr8\" (UniqueName: \"kubernetes.io/projected/357d9f32-903d-472b-bf56-f5151ba1ca8b-kube-api-access-xrnr8\") pod \"cluster-samples-operator-79b7c55f87-kv62p\" (UID: \"357d9f32-903d-472b-bf56-f5151ba1ca8b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-kv62p"
Jul 23 14:56:44.394462 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:56:44.394337    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a253789c-c1c0-409f-b335-63fe5f2fb4af-tmp\") pod \"insights-operator-555d9d7885-zndrp\" (UID: \"a253789c-c1c0-409f-b335-63fe5f2fb4af\") " pod="openshift-insights/insights-operator-555d9d7885-zndrp"
Jul 23 14:56:44.405201 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:56:44.405173    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrnr8\" (UniqueName: \"kubernetes.io/projected/357d9f32-903d-472b-bf56-f5151ba1ca8b-kube-api-access-xrnr8\") pod \"cluster-samples-operator-79b7c55f87-kv62p\" (UID: \"357d9f32-903d-472b-bf56-f5151ba1ca8b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-kv62p"
Jul 23 14:56:44.495069 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:56:44.494978    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/a253789c-c1c0-409f-b335-63fe5f2fb4af-snapshots\") pod \"insights-operator-555d9d7885-zndrp\" (UID: \"a253789c-c1c0-409f-b335-63fe5f2fb4af\") " pod="openshift-insights/insights-operator-555d9d7885-zndrp"
Jul 23 14:56:44.495069 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:56:44.495018    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7jdh\" (UniqueName: \"kubernetes.io/projected/a253789c-c1c0-409f-b335-63fe5f2fb4af-kube-api-access-w7jdh\") pod \"insights-operator-555d9d7885-zndrp\" (UID: \"a253789c-c1c0-409f-b335-63fe5f2fb4af\") " pod="openshift-insights/insights-operator-555d9d7885-zndrp"
Jul 23 14:56:44.495069 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:56:44.495042    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a253789c-c1c0-409f-b335-63fe5f2fb4af-tmp\") pod \"insights-operator-555d9d7885-zndrp\" (UID: \"a253789c-c1c0-409f-b335-63fe5f2fb4af\") " pod="openshift-insights/insights-operator-555d9d7885-zndrp"
Jul 23 14:56:44.495320 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:56:44.495171    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a253789c-c1c0-409f-b335-63fe5f2fb4af-serving-cert\") pod \"insights-operator-555d9d7885-zndrp\" (UID: \"a253789c-c1c0-409f-b335-63fe5f2fb4af\") " pod="openshift-insights/insights-operator-555d9d7885-zndrp"
Jul 23 14:56:44.495320 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:56:44.495208    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7df58b86-f921-4316-b8b1-f48137124a76-image-registry-private-configuration\") pod \"image-registry-5cf48ff8c9-p9xlr\" (UID: \"7df58b86-f921-4316-b8b1-f48137124a76\") " pod="openshift-image-registry/image-registry-5cf48ff8c9-p9xlr"
Jul 23 14:56:44.495320 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:56:44.495226    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7df58b86-f921-4316-b8b1-f48137124a76-bound-sa-token\") pod \"image-registry-5cf48ff8c9-p9xlr\" (UID: \"7df58b86-f921-4316-b8b1-f48137124a76\") " pod="openshift-image-registry/image-registry-5cf48ff8c9-p9xlr"
Jul 23 14:56:44.495320 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:56:44.495287    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7df58b86-f921-4316-b8b1-f48137124a76-trusted-ca\") pod \"image-registry-5cf48ff8c9-p9xlr\" (UID: \"7df58b86-f921-4316-b8b1-f48137124a76\") " pod="openshift-image-registry/image-registry-5cf48ff8c9-p9xlr"
Jul 23 14:56:44.495320 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:56:44.495315    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9ae597eb-68e0-447c-bf0c-781a80684af1-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-k5zd5\" (UID: \"9ae597eb-68e0-447c-bf0c-781a80684af1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-k5zd5"
Jul 23 14:56:44.495568 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:56:44.495343    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7df58b86-f921-4316-b8b1-f48137124a76-installation-pull-secrets\") pod \"image-registry-5cf48ff8c9-p9xlr\" (UID: \"7df58b86-f921-4316-b8b1-f48137124a76\") " pod="openshift-image-registry/image-registry-5cf48ff8c9-p9xlr"
Jul 23 14:56:44.495568 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:56:44.495368    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fjj5\" (UniqueName: \"kubernetes.io/projected/7df58b86-f921-4316-b8b1-f48137124a76-kube-api-access-9fjj5\") pod \"image-registry-5cf48ff8c9-p9xlr\" (UID: \"7df58b86-f921-4316-b8b1-f48137124a76\") " pod="openshift-image-registry/image-registry-5cf48ff8c9-p9xlr"
Jul 23 14:56:44.495686 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:56:44.495653    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a253789c-c1c0-409f-b335-63fe5f2fb4af-tmp\") pod \"insights-operator-555d9d7885-zndrp\" (UID: \"a253789c-c1c0-409f-b335-63fe5f2fb4af\") " pod="openshift-insights/insights-operator-555d9d7885-zndrp"
Jul 23 14:56:44.495735 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:56:44.495700    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7df58b86-f921-4316-b8b1-f48137124a76-registry-tls\") pod \"image-registry-5cf48ff8c9-p9xlr\" (UID: \"7df58b86-f921-4316-b8b1-f48137124a76\") " pod="openshift-image-registry/image-registry-5cf48ff8c9-p9xlr"
Jul 23 14:56:44.495735 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:56:44.495728    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7df58b86-f921-4316-b8b1-f48137124a76-registry-certificates\") pod \"image-registry-5cf48ff8c9-p9xlr\" (UID: \"7df58b86-f921-4316-b8b1-f48137124a76\") " pod="openshift-image-registry/image-registry-5cf48ff8c9-p9xlr"
Jul 23 14:56:44.495896 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:56:44.495760    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrwcp\" (UniqueName: \"kubernetes.io/projected/9ae597eb-68e0-447c-bf0c-781a80684af1-kube-api-access-jrwcp\") pod \"kube-storage-version-migrator-operator-7658d9b749-k5zd5\" (UID: \"9ae597eb-68e0-447c-bf0c-781a80684af1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-k5zd5"
Jul 23 14:56:44.495896 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:56:44.495778    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/a253789c-c1c0-409f-b335-63fe5f2fb4af-snapshots\") pod \"insights-operator-555d9d7885-zndrp\" (UID: \"a253789c-c1c0-409f-b335-63fe5f2fb4af\") " pod="openshift-insights/insights-operator-555d9d7885-zndrp"
Jul 23 14:56:44.495896 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:56:44.495789    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9ae597eb-68e0-447c-bf0c-781a80684af1-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-k5zd5\" (UID: \"9ae597eb-68e0-447c-bf0c-781a80684af1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-k5zd5"
Jul 23 14:56:44.495896 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:56:44.495837    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7df58b86-f921-4316-b8b1-f48137124a76-ca-trust-extracted\") pod \"image-registry-5cf48ff8c9-p9xlr\" (UID: \"7df58b86-f921-4316-b8b1-f48137124a76\") " pod="openshift-image-registry/image-registry-5cf48ff8c9-p9xlr"
Jul 23 14:56:44.495896 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:56:44.495870    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a253789c-c1c0-409f-b335-63fe5f2fb4af-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-zndrp\" (UID: \"a253789c-c1c0-409f-b335-63fe5f2fb4af\") " pod="openshift-insights/insights-operator-555d9d7885-zndrp"
Jul 23 14:56:44.495896 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:56:44.495878    2560 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 23 14:56:44.495896 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:56:44.495896    2560 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5cf48ff8c9-p9xlr: secret "image-registry-tls" not found
Jul 23 14:56:44.496269 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:56:44.495968    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7df58b86-f921-4316-b8b1-f48137124a76-registry-tls podName:7df58b86-f921-4316-b8b1-f48137124a76 nodeName:}" failed. No retries permitted until 2026-07-23 14:56:44.995946847 +0000 UTC m=+246.503820573 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7df58b86-f921-4316-b8b1-f48137124a76-registry-tls") pod "image-registry-5cf48ff8c9-p9xlr" (UID: "7df58b86-f921-4316-b8b1-f48137124a76") : secret "image-registry-tls" not found
Jul 23 14:56:44.496269 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:56:44.495987    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a253789c-c1c0-409f-b335-63fe5f2fb4af-service-ca-bundle\") pod \"insights-operator-555d9d7885-zndrp\" (UID: \"a253789c-c1c0-409f-b335-63fe5f2fb4af\") " pod="openshift-insights/insights-operator-555d9d7885-zndrp"
Jul 23 14:56:44.496392 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:56:44.496292    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9ae597eb-68e0-447c-bf0c-781a80684af1-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-k5zd5\" (UID: \"9ae597eb-68e0-447c-bf0c-781a80684af1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-k5zd5"
Jul 23 14:56:44.496392 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:56:44.496343    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7df58b86-f921-4316-b8b1-f48137124a76-trusted-ca\") pod \"image-registry-5cf48ff8c9-p9xlr\" (UID: \"7df58b86-f921-4316-b8b1-f48137124a76\") " pod="openshift-image-registry/image-registry-5cf48ff8c9-p9xlr"
Jul 23 14:56:44.496839 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:56:44.496714    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a253789c-c1c0-409f-b335-63fe5f2fb4af-service-ca-bundle\") pod \"insights-operator-555d9d7885-zndrp\" (UID: \"a253789c-c1c0-409f-b335-63fe5f2fb4af\") " pod="openshift-insights/insights-operator-555d9d7885-zndrp"
Jul 23 14:56:44.496839 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:56:44.496788    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7df58b86-f921-4316-b8b1-f48137124a76-registry-certificates\") pod \"image-registry-5cf48ff8c9-p9xlr\" (UID: \"7df58b86-f921-4316-b8b1-f48137124a76\") " pod="openshift-image-registry/image-registry-5cf48ff8c9-p9xlr"
Jul 23 14:56:44.497011 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:56:44.496958    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7df58b86-f921-4316-b8b1-f48137124a76-ca-trust-extracted\") pod \"image-registry-5cf48ff8c9-p9xlr\" (UID: \"7df58b86-f921-4316-b8b1-f48137124a76\") " pod="openshift-image-registry/image-registry-5cf48ff8c9-p9xlr"
Jul 23 14:56:44.497011 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:56:44.496999    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a253789c-c1c0-409f-b335-63fe5f2fb4af-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-zndrp\" (UID: \"a253789c-c1c0-409f-b335-63fe5f2fb4af\") " pod="openshift-insights/insights-operator-555d9d7885-zndrp"
Jul 23 14:56:44.498034 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:56:44.498010    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a253789c-c1c0-409f-b335-63fe5f2fb4af-serving-cert\") pod \"insights-operator-555d9d7885-zndrp\" (UID: \"a253789c-c1c0-409f-b335-63fe5f2fb4af\") " pod="openshift-insights/insights-operator-555d9d7885-zndrp"
Jul 23 14:56:44.498127 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:56:44.498041    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7df58b86-f921-4316-b8b1-f48137124a76-image-registry-private-configuration\") pod \"image-registry-5cf48ff8c9-p9xlr\" (UID: \"7df58b86-f921-4316-b8b1-f48137124a76\") " pod="openshift-image-registry/image-registry-5cf48ff8c9-p9xlr"
Jul 23 14:56:44.498374 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:56:44.498358    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7df58b86-f921-4316-b8b1-f48137124a76-installation-pull-secrets\") pod \"image-registry-5cf48ff8c9-p9xlr\" (UID: \"7df58b86-f921-4316-b8b1-f48137124a76\") " pod="openshift-image-registry/image-registry-5cf48ff8c9-p9xlr"
Jul 23 14:56:44.498711 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:56:44.498692    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9ae597eb-68e0-447c-bf0c-781a80684af1-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-k5zd5\" (UID: \"9ae597eb-68e0-447c-bf0c-781a80684af1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-k5zd5"
Jul 23 14:56:44.505827 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:56:44.505778    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7jdh\" (UniqueName: \"kubernetes.io/projected/a253789c-c1c0-409f-b335-63fe5f2fb4af-kube-api-access-w7jdh\") pod \"insights-operator-555d9d7885-zndrp\" (UID: \"a253789c-c1c0-409f-b335-63fe5f2fb4af\") " pod="openshift-insights/insights-operator-555d9d7885-zndrp"
Jul 23 14:56:44.505931 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:56:44.505841    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrwcp\" (UniqueName: \"kubernetes.io/projected/9ae597eb-68e0-447c-bf0c-781a80684af1-kube-api-access-jrwcp\") pod \"kube-storage-version-migrator-operator-7658d9b749-k5zd5\" (UID: \"9ae597eb-68e0-447c-bf0c-781a80684af1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-k5zd5"
Jul 23 14:56:44.506168 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:56:44.506152    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7df58b86-f921-4316-b8b1-f48137124a76-bound-sa-token\") pod \"image-registry-5cf48ff8c9-p9xlr\" (UID: \"7df58b86-f921-4316-b8b1-f48137124a76\") " pod="openshift-image-registry/image-registry-5cf48ff8c9-p9xlr"
Jul 23 14:56:44.506442 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:56:44.506423    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fjj5\" (UniqueName: \"kubernetes.io/projected/7df58b86-f921-4316-b8b1-f48137124a76-kube-api-access-9fjj5\") pod \"image-registry-5cf48ff8c9-p9xlr\" (UID: \"7df58b86-f921-4316-b8b1-f48137124a76\") " pod="openshift-image-registry/image-registry-5cf48ff8c9-p9xlr"
Jul 23 14:56:44.627324 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:56:44.627283    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-k5zd5"
Jul 23 14:56:44.635536 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:56:44.635516    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-555d9d7885-zndrp"
Jul 23 14:56:44.757522 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:56:44.757430    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-k5zd5"]
Jul 23 14:56:44.760127 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:56:44.760075    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ae597eb_68e0_447c_bf0c_781a80684af1.slice/crio-6bdf2d94a339f77ea5597d8356cb9905403dddbafb248299d9fd5700971fffb7 WatchSource:0}: Error finding container 6bdf2d94a339f77ea5597d8356cb9905403dddbafb248299d9fd5700971fffb7: Status 404 returned error can't find the container with id 6bdf2d94a339f77ea5597d8356cb9905403dddbafb248299d9fd5700971fffb7
Jul 23 14:56:44.774828 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:56:44.774800    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-555d9d7885-zndrp"]
Jul 23 14:56:44.778091 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:56:44.778071    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda253789c_c1c0_409f_b335_63fe5f2fb4af.slice/crio-7b4d0531a7295b8147f3c0f500247b76abf822ede567e897529a58af178d66e9 WatchSource:0}: Error finding container 7b4d0531a7295b8147f3c0f500247b76abf822ede567e897529a58af178d66e9: Status 404 returned error can't find the container with id 7b4d0531a7295b8147f3c0f500247b76abf822ede567e897529a58af178d66e9
Jul 23 14:56:44.813139 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:56:44.813107    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-k5zd5" event={"ID":"9ae597eb-68e0-447c-bf0c-781a80684af1","Type":"ContainerStarted","Data":"6bdf2d94a339f77ea5597d8356cb9905403dddbafb248299d9fd5700971fffb7"}
Jul 23 14:56:44.814371 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:56:44.814339    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-zndrp" event={"ID":"a253789c-c1c0-409f-b335-63fe5f2fb4af","Type":"ContainerStarted","Data":"7b4d0531a7295b8147f3c0f500247b76abf822ede567e897529a58af178d66e9"}
Jul 23 14:56:44.900392 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:56:44.900353    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/357d9f32-903d-472b-bf56-f5151ba1ca8b-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-kv62p\" (UID: \"357d9f32-903d-472b-bf56-f5151ba1ca8b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-kv62p"
Jul 23 14:56:44.900534 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:56:44.900497    2560 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 23 14:56:44.900574 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:56:44.900561    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/357d9f32-903d-472b-bf56-f5151ba1ca8b-samples-operator-tls podName:357d9f32-903d-472b-bf56-f5151ba1ca8b nodeName:}" failed. No retries permitted until 2026-07-23 14:56:45.900544167 +0000 UTC m=+247.408417894 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/357d9f32-903d-472b-bf56-f5151ba1ca8b-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-kv62p" (UID: "357d9f32-903d-472b-bf56-f5151ba1ca8b") : secret "samples-operator-tls" not found
Jul 23 14:56:45.000753 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:56:45.000715    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7df58b86-f921-4316-b8b1-f48137124a76-registry-tls\") pod \"image-registry-5cf48ff8c9-p9xlr\" (UID: \"7df58b86-f921-4316-b8b1-f48137124a76\") " pod="openshift-image-registry/image-registry-5cf48ff8c9-p9xlr"
Jul 23 14:56:45.000880 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:56:45.000864    2560 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 23 14:56:45.000934 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:56:45.000880    2560 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5cf48ff8c9-p9xlr: secret "image-registry-tls" not found
Jul 23 14:56:45.000934 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:56:45.000932    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7df58b86-f921-4316-b8b1-f48137124a76-registry-tls podName:7df58b86-f921-4316-b8b1-f48137124a76 nodeName:}" failed. No retries permitted until 2026-07-23 14:56:46.000916463 +0000 UTC m=+247.508790209 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7df58b86-f921-4316-b8b1-f48137124a76-registry-tls") pod "image-registry-5cf48ff8c9-p9xlr" (UID: "7df58b86-f921-4316-b8b1-f48137124a76") : secret "image-registry-tls" not found
Jul 23 14:56:45.908812 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:56:45.908779    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/357d9f32-903d-472b-bf56-f5151ba1ca8b-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-kv62p\" (UID: \"357d9f32-903d-472b-bf56-f5151ba1ca8b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-kv62p"
Jul 23 14:56:45.909295 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:56:45.908934    2560 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 23 14:56:45.909295 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:56:45.909026    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/357d9f32-903d-472b-bf56-f5151ba1ca8b-samples-operator-tls podName:357d9f32-903d-472b-bf56-f5151ba1ca8b nodeName:}" failed. No retries permitted until 2026-07-23 14:56:47.909007775 +0000 UTC m=+249.416881502 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/357d9f32-903d-472b-bf56-f5151ba1ca8b-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-kv62p" (UID: "357d9f32-903d-472b-bf56-f5151ba1ca8b") : secret "samples-operator-tls" not found
Jul 23 14:56:46.009756 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:56:46.009684    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7df58b86-f921-4316-b8b1-f48137124a76-registry-tls\") pod \"image-registry-5cf48ff8c9-p9xlr\" (UID: \"7df58b86-f921-4316-b8b1-f48137124a76\") " pod="openshift-image-registry/image-registry-5cf48ff8c9-p9xlr"
Jul 23 14:56:46.009966 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:56:46.009844    2560 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 23 14:56:46.009966 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:56:46.009863    2560 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5cf48ff8c9-p9xlr: secret "image-registry-tls" not found
Jul 23 14:56:46.009966 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:56:46.009928    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7df58b86-f921-4316-b8b1-f48137124a76-registry-tls podName:7df58b86-f921-4316-b8b1-f48137124a76 nodeName:}" failed. No retries permitted until 2026-07-23 14:56:48.009909917 +0000 UTC m=+249.517783647 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7df58b86-f921-4316-b8b1-f48137124a76-registry-tls") pod "image-registry-5cf48ff8c9-p9xlr" (UID: "7df58b86-f921-4316-b8b1-f48137124a76") : secret "image-registry-tls" not found
Jul 23 14:56:47.826987 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:56:47.826930    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-k5zd5" event={"ID":"9ae597eb-68e0-447c-bf0c-781a80684af1","Type":"ContainerStarted","Data":"aa8036d24136eb896ca9348b78a1672d041699f8716fd9f3eb00fdeaf51688d1"}
Jul 23 14:56:47.828429 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:56:47.828397    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-zndrp" event={"ID":"a253789c-c1c0-409f-b335-63fe5f2fb4af","Type":"ContainerStarted","Data":"04382b3482c980cae213d693d100204db33b50bd25cf42774bc181b7ec648ed3"}
Jul 23 14:56:47.844036 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:56:47.843894    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-k5zd5" podStartSLOduration=1.453091565 podStartE2EDuration="3.843876678s" totalImagesPullingTime="2.390785113s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 14:56:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 14:56:47.843429308 +0000 UTC m=+249.351303057" watchObservedRunningTime="2026-07-23 14:56:47.843876678 +0000 UTC m=+249.351750428"
Jul 23 14:56:47.860425 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:56:47.860387    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-555d9d7885-zndrp" podStartSLOduration=1.4901718210000001 podStartE2EDuration="3.86036935s" totalImagesPullingTime="2.370197529s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 14:56:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 14:56:47.859620615 +0000 UTC m=+249.367494368" watchObservedRunningTime="2026-07-23 14:56:47.86036935 +0000 UTC m=+249.368243098"
Jul 23 14:56:47.924354 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:56:47.923766    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/357d9f32-903d-472b-bf56-f5151ba1ca8b-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-kv62p\" (UID: \"357d9f32-903d-472b-bf56-f5151ba1ca8b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-kv62p"
Jul 23 14:56:47.924354 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:56:47.924209    2560 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 23 14:56:47.924354 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:56:47.924314    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/357d9f32-903d-472b-bf56-f5151ba1ca8b-samples-operator-tls podName:357d9f32-903d-472b-bf56-f5151ba1ca8b nodeName:}" failed. No retries permitted until 2026-07-23 14:56:51.924294409 +0000 UTC m=+253.432168139 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/357d9f32-903d-472b-bf56-f5151ba1ca8b-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-kv62p" (UID: "357d9f32-903d-472b-bf56-f5151ba1ca8b") : secret "samples-operator-tls" not found
Jul 23 14:56:48.026005 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:56:48.025534    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7df58b86-f921-4316-b8b1-f48137124a76-registry-tls\") pod \"image-registry-5cf48ff8c9-p9xlr\" (UID: \"7df58b86-f921-4316-b8b1-f48137124a76\") " pod="openshift-image-registry/image-registry-5cf48ff8c9-p9xlr"
Jul 23 14:56:48.026005 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:56:48.025733    2560 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 23 14:56:48.026005 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:56:48.025751    2560 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5cf48ff8c9-p9xlr: secret "image-registry-tls" not found
Jul 23 14:56:48.026005 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:56:48.025808    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7df58b86-f921-4316-b8b1-f48137124a76-registry-tls podName:7df58b86-f921-4316-b8b1-f48137124a76 nodeName:}" failed. No retries permitted until 2026-07-23 14:56:52.025789158 +0000 UTC m=+253.533662886 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7df58b86-f921-4316-b8b1-f48137124a76-registry-tls") pod "image-registry-5cf48ff8c9-p9xlr" (UID: "7df58b86-f921-4316-b8b1-f48137124a76") : secret "image-registry-tls" not found
Jul 23 14:56:48.119865 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:56:48.119793    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-5xjtm"]
Jul 23 14:56:48.122015 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:56:48.121994    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-5xjtm"
Jul 23 14:56:48.124966 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:56:48.124949    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 23 14:56:48.125307 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:56:48.125288    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 23 14:56:48.125597 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:56:48.125581    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-pmdlz\""
Jul 23 14:56:48.136173 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:56:48.136144    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-5xjtm"]
Jul 23 14:56:48.226705 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:56:48.226669    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7l92x\" (UniqueName: \"kubernetes.io/projected/823379c6-9443-4acd-9c9f-8c300490b6b2-kube-api-access-7l92x\") pod \"insights-runtime-extractor-5xjtm\" (UID: \"823379c6-9443-4acd-9c9f-8c300490b6b2\") " pod="openshift-insights/insights-runtime-extractor-5xjtm"
Jul 23 14:56:48.226866 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:56:48.226725    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/823379c6-9443-4acd-9c9f-8c300490b6b2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5xjtm\" (UID: \"823379c6-9443-4acd-9c9f-8c300490b6b2\") " pod="openshift-insights/insights-runtime-extractor-5xjtm"
Jul 23 14:56:48.226866 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:56:48.226796    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/823379c6-9443-4acd-9c9f-8c300490b6b2-crio-socket\") pod \"insights-runtime-extractor-5xjtm\" (UID: \"823379c6-9443-4acd-9c9f-8c300490b6b2\") " pod="openshift-insights/insights-runtime-extractor-5xjtm"
Jul 23 14:56:48.226866 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:56:48.226840    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/823379c6-9443-4acd-9c9f-8c300490b6b2-data-volume\") pod \"insights-runtime-extractor-5xjtm\" (UID: \"823379c6-9443-4acd-9c9f-8c300490b6b2\") " pod="openshift-insights/insights-runtime-extractor-5xjtm"
Jul 23 14:56:48.226866 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:56:48.226860    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/823379c6-9443-4acd-9c9f-8c300490b6b2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5xjtm\" (UID: \"823379c6-9443-4acd-9c9f-8c300490b6b2\") " pod="openshift-insights/insights-runtime-extractor-5xjtm"
Jul 23 14:56:48.327508 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:56:48.327470    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/823379c6-9443-4acd-9c9f-8c300490b6b2-crio-socket\") pod \"insights-runtime-extractor-5xjtm\" (UID: \"823379c6-9443-4acd-9c9f-8c300490b6b2\") " pod="openshift-insights/insights-runtime-extractor-5xjtm"
Jul 23 14:56:48.327710 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:56:48.327540    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/823379c6-9443-4acd-9c9f-8c300490b6b2-data-volume\") pod \"insights-runtime-extractor-5xjtm\" (UID: \"823379c6-9443-4acd-9c9f-8c300490b6b2\") " pod="openshift-insights/insights-runtime-extractor-5xjtm"
Jul 23 14:56:48.327710 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:56:48.327574    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/823379c6-9443-4acd-9c9f-8c300490b6b2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5xjtm\" (UID: \"823379c6-9443-4acd-9c9f-8c300490b6b2\") " pod="openshift-insights/insights-runtime-extractor-5xjtm"
Jul 23 14:56:48.327710 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:56:48.327614    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/823379c6-9443-4acd-9c9f-8c300490b6b2-crio-socket\") pod \"insights-runtime-extractor-5xjtm\" (UID: \"823379c6-9443-4acd-9c9f-8c300490b6b2\") " pod="openshift-insights/insights-runtime-extractor-5xjtm"
Jul 23 14:56:48.327710 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:56:48.327653    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7l92x\" (UniqueName: \"kubernetes.io/projected/823379c6-9443-4acd-9c9f-8c300490b6b2-kube-api-access-7l92x\") pod \"insights-runtime-extractor-5xjtm\" (UID: \"823379c6-9443-4acd-9c9f-8c300490b6b2\") " pod="openshift-insights/insights-runtime-extractor-5xjtm"
Jul 23 14:56:48.327710 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:56:48.327694    2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 23 14:56:48.327710 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:56:48.327703    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/823379c6-9443-4acd-9c9f-8c300490b6b2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5xjtm\" (UID: \"823379c6-9443-4acd-9c9f-8c300490b6b2\") " pod="openshift-insights/insights-runtime-extractor-5xjtm"
Jul 23 14:56:48.327976 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:56:48.327763    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/823379c6-9443-4acd-9c9f-8c300490b6b2-kube-rbac-proxy-cm podName:823379c6-9443-4acd-9c9f-8c300490b6b2 nodeName:}" failed. No retries permitted until 2026-07-23 14:56:48.827743515 +0000 UTC m=+250.335617261 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/823379c6-9443-4acd-9c9f-8c300490b6b2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5xjtm" (UID: "823379c6-9443-4acd-9c9f-8c300490b6b2") : configmap "kube-rbac-proxy" not found
Jul 23 14:56:48.327976 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:56:48.327811    2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 23 14:56:48.327976 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:56:48.327860    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/823379c6-9443-4acd-9c9f-8c300490b6b2-insights-runtime-extractor-tls podName:823379c6-9443-4acd-9c9f-8c300490b6b2 nodeName:}" failed. No retries permitted until 2026-07-23 14:56:48.827844442 +0000 UTC m=+250.335718174 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/823379c6-9443-4acd-9c9f-8c300490b6b2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5xjtm" (UID: "823379c6-9443-4acd-9c9f-8c300490b6b2") : secret "insights-runtime-extractor-tls" not found
Jul 23 14:56:48.327976 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:56:48.327891    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/823379c6-9443-4acd-9c9f-8c300490b6b2-data-volume\") pod \"insights-runtime-extractor-5xjtm\" (UID: \"823379c6-9443-4acd-9c9f-8c300490b6b2\") " pod="openshift-insights/insights-runtime-extractor-5xjtm"
Jul 23 14:56:48.335922 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:56:48.335893    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7l92x\" (UniqueName: \"kubernetes.io/projected/823379c6-9443-4acd-9c9f-8c300490b6b2-kube-api-access-7l92x\") pod \"insights-runtime-extractor-5xjtm\" (UID: \"823379c6-9443-4acd-9c9f-8c300490b6b2\") " pod="openshift-insights/insights-runtime-extractor-5xjtm"
Jul 23 14:56:48.831196 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:56:48.831158    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/823379c6-9443-4acd-9c9f-8c300490b6b2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5xjtm\" (UID: \"823379c6-9443-4acd-9c9f-8c300490b6b2\") " pod="openshift-insights/insights-runtime-extractor-5xjtm"
Jul 23 14:56:48.831631 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:56:48.831256    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/823379c6-9443-4acd-9c9f-8c300490b6b2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5xjtm\" (UID: \"823379c6-9443-4acd-9c9f-8c300490b6b2\") " pod="openshift-insights/insights-runtime-extractor-5xjtm"
Jul 23 14:56:48.831631 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:56:48.831302    2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 23 14:56:48.831631 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:56:48.831369    2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 23 14:56:48.831631 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:56:48.831378    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/823379c6-9443-4acd-9c9f-8c300490b6b2-kube-rbac-proxy-cm podName:823379c6-9443-4acd-9c9f-8c300490b6b2 nodeName:}" failed. No retries permitted until 2026-07-23 14:56:49.831358564 +0000 UTC m=+251.339232295 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/823379c6-9443-4acd-9c9f-8c300490b6b2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5xjtm" (UID: "823379c6-9443-4acd-9c9f-8c300490b6b2") : configmap "kube-rbac-proxy" not found
Jul 23 14:56:48.831631 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:56:48.831424    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/823379c6-9443-4acd-9c9f-8c300490b6b2-insights-runtime-extractor-tls podName:823379c6-9443-4acd-9c9f-8c300490b6b2 nodeName:}" failed. No retries permitted until 2026-07-23 14:56:49.831406135 +0000 UTC m=+251.339279861 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/823379c6-9443-4acd-9c9f-8c300490b6b2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5xjtm" (UID: "823379c6-9443-4acd-9c9f-8c300490b6b2") : secret "insights-runtime-extractor-tls" not found
Jul 23 14:56:49.840945 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:56:49.840914    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/823379c6-9443-4acd-9c9f-8c300490b6b2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5xjtm\" (UID: \"823379c6-9443-4acd-9c9f-8c300490b6b2\") " pod="openshift-insights/insights-runtime-extractor-5xjtm"
Jul 23 14:56:49.841448 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:56:49.840962    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/823379c6-9443-4acd-9c9f-8c300490b6b2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5xjtm\" (UID: \"823379c6-9443-4acd-9c9f-8c300490b6b2\") " pod="openshift-insights/insights-runtime-extractor-5xjtm"
Jul 23 14:56:49.841448 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:56:49.841049    2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 23 14:56:49.841448 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:56:49.841049    2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 23 14:56:49.841448 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:56:49.841098    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/823379c6-9443-4acd-9c9f-8c300490b6b2-insights-runtime-extractor-tls podName:823379c6-9443-4acd-9c9f-8c300490b6b2 nodeName:}" failed. No retries permitted until 2026-07-23 14:56:51.841084511 +0000 UTC m=+253.348958238 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/823379c6-9443-4acd-9c9f-8c300490b6b2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5xjtm" (UID: "823379c6-9443-4acd-9c9f-8c300490b6b2") : secret "insights-runtime-extractor-tls" not found
Jul 23 14:56:49.841448 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:56:49.841111    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/823379c6-9443-4acd-9c9f-8c300490b6b2-kube-rbac-proxy-cm podName:823379c6-9443-4acd-9c9f-8c300490b6b2 nodeName:}" failed. No retries permitted until 2026-07-23 14:56:51.841104855 +0000 UTC m=+253.348978582 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/823379c6-9443-4acd-9c9f-8c300490b6b2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5xjtm" (UID: "823379c6-9443-4acd-9c9f-8c300490b6b2") : configmap "kube-rbac-proxy" not found
Jul 23 14:56:50.289626 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:56:50.289602    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-vfsh6_d9b3550d-e82f-4934-b093-486563f240fb/dns-node-resolver/0.log"
Jul 23 14:56:50.748927 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:56:50.748895    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/413d2464-7258-4246-85bb-ac9f03976ee5-metrics-certs\") pod \"network-metrics-daemon-hvw8v\" (UID: \"413d2464-7258-4246-85bb-ac9f03976ee5\") " pod="openshift-multus/network-metrics-daemon-hvw8v"
Jul 23 14:56:50.749093 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:56:50.749053    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 23 14:56:50.749132 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:56:50.749116    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/413d2464-7258-4246-85bb-ac9f03976ee5-metrics-certs podName:413d2464-7258-4246-85bb-ac9f03976ee5 nodeName:}" failed. No retries permitted until 2026-07-23 14:58:52.749101474 +0000 UTC m=+374.256975201 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/413d2464-7258-4246-85bb-ac9f03976ee5-metrics-certs") pod "network-metrics-daemon-hvw8v" (UID: "413d2464-7258-4246-85bb-ac9f03976ee5") : secret "metrics-daemon-secret" not found
Jul 23 14:56:51.289651 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:56:51.289620    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-lz8qd_9a2b87d6-d3dd-41b8-96f8-2e7a0be09325/node-ca/0.log"
Jul 23 14:56:51.855272 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:56:51.855212    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/823379c6-9443-4acd-9c9f-8c300490b6b2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5xjtm\" (UID: \"823379c6-9443-4acd-9c9f-8c300490b6b2\") " pod="openshift-insights/insights-runtime-extractor-5xjtm"
Jul 23 14:56:51.855468 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:56:51.855327    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/823379c6-9443-4acd-9c9f-8c300490b6b2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5xjtm\" (UID: \"823379c6-9443-4acd-9c9f-8c300490b6b2\") " pod="openshift-insights/insights-runtime-extractor-5xjtm"
Jul 23 14:56:51.855468 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:56:51.855379    2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 23 14:56:51.855468 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:56:51.855438    2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 23 14:56:51.855646 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:56:51.855444    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/823379c6-9443-4acd-9c9f-8c300490b6b2-insights-runtime-extractor-tls podName:823379c6-9443-4acd-9c9f-8c300490b6b2 nodeName:}" failed. No retries permitted until 2026-07-23 14:56:55.855428389 +0000 UTC m=+257.363302115 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/823379c6-9443-4acd-9c9f-8c300490b6b2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5xjtm" (UID: "823379c6-9443-4acd-9c9f-8c300490b6b2") : secret "insights-runtime-extractor-tls" not found
Jul 23 14:56:51.855646 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:56:51.855491    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/823379c6-9443-4acd-9c9f-8c300490b6b2-kube-rbac-proxy-cm podName:823379c6-9443-4acd-9c9f-8c300490b6b2 nodeName:}" failed. No retries permitted until 2026-07-23 14:56:55.855478511 +0000 UTC m=+257.363352237 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/823379c6-9443-4acd-9c9f-8c300490b6b2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5xjtm" (UID: "823379c6-9443-4acd-9c9f-8c300490b6b2") : configmap "kube-rbac-proxy" not found
Jul 23 14:56:51.956572 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:56:51.956535    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/357d9f32-903d-472b-bf56-f5151ba1ca8b-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-kv62p\" (UID: \"357d9f32-903d-472b-bf56-f5151ba1ca8b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-kv62p"
Jul 23 14:56:51.956739 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:56:51.956674    2560 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 23 14:56:51.956739 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:56:51.956739    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/357d9f32-903d-472b-bf56-f5151ba1ca8b-samples-operator-tls podName:357d9f32-903d-472b-bf56-f5151ba1ca8b nodeName:}" failed. No retries permitted until 2026-07-23 14:56:59.956724929 +0000 UTC m=+261.464598654 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/357d9f32-903d-472b-bf56-f5151ba1ca8b-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-kv62p" (UID: "357d9f32-903d-472b-bf56-f5151ba1ca8b") : secret "samples-operator-tls" not found
Jul 23 14:56:52.057602 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:56:52.057561    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7df58b86-f921-4316-b8b1-f48137124a76-registry-tls\") pod \"image-registry-5cf48ff8c9-p9xlr\" (UID: \"7df58b86-f921-4316-b8b1-f48137124a76\") " pod="openshift-image-registry/image-registry-5cf48ff8c9-p9xlr"
Jul 23 14:56:52.057779 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:56:52.057703    2560 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 23 14:56:52.057779 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:56:52.057719    2560 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5cf48ff8c9-p9xlr: secret "image-registry-tls" not found
Jul 23 14:56:52.057779 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:56:52.057770    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7df58b86-f921-4316-b8b1-f48137124a76-registry-tls podName:7df58b86-f921-4316-b8b1-f48137124a76 nodeName:}" failed. No retries permitted until 2026-07-23 14:57:00.057754902 +0000 UTC m=+261.565628629 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7df58b86-f921-4316-b8b1-f48137124a76-registry-tls") pod "image-registry-5cf48ff8c9-p9xlr" (UID: "7df58b86-f921-4316-b8b1-f48137124a76") : secret "image-registry-tls" not found
Jul 23 14:56:52.092196 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:56:52.092168    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-sw5j8_f7786ed8-2ad5-40ad-a866-ddacdead5ab1/serve-healthcheck-canary/0.log"
Jul 23 14:56:55.884685 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:56:55.884652    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/823379c6-9443-4acd-9c9f-8c300490b6b2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5xjtm\" (UID: \"823379c6-9443-4acd-9c9f-8c300490b6b2\") " pod="openshift-insights/insights-runtime-extractor-5xjtm"
Jul 23 14:56:55.885122 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:56:55.884712    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/823379c6-9443-4acd-9c9f-8c300490b6b2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5xjtm\" (UID: \"823379c6-9443-4acd-9c9f-8c300490b6b2\") " pod="openshift-insights/insights-runtime-extractor-5xjtm"
Jul 23 14:56:55.885122 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:56:55.884810    2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 23 14:56:55.885122 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:56:55.884826    2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 23 14:56:55.885122 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:56:55.884871    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/823379c6-9443-4acd-9c9f-8c300490b6b2-insights-runtime-extractor-tls podName:823379c6-9443-4acd-9c9f-8c300490b6b2 nodeName:}" failed. No retries permitted until 2026-07-23 14:57:03.884853259 +0000 UTC m=+265.392726986 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/823379c6-9443-4acd-9c9f-8c300490b6b2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5xjtm" (UID: "823379c6-9443-4acd-9c9f-8c300490b6b2") : secret "insights-runtime-extractor-tls" not found
Jul 23 14:56:55.885122 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:56:55.884887    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/823379c6-9443-4acd-9c9f-8c300490b6b2-kube-rbac-proxy-cm podName:823379c6-9443-4acd-9c9f-8c300490b6b2 nodeName:}" failed. No retries permitted until 2026-07-23 14:57:03.884880491 +0000 UTC m=+265.392754217 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/823379c6-9443-4acd-9c9f-8c300490b6b2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5xjtm" (UID: "823379c6-9443-4acd-9c9f-8c300490b6b2") : configmap "kube-rbac-proxy" not found
Jul 23 14:57:00.020631 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:00.020550    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/357d9f32-903d-472b-bf56-f5151ba1ca8b-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-kv62p\" (UID: \"357d9f32-903d-472b-bf56-f5151ba1ca8b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-kv62p"
Jul 23 14:57:00.023004 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:00.022983    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/357d9f32-903d-472b-bf56-f5151ba1ca8b-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-kv62p\" (UID: \"357d9f32-903d-472b-bf56-f5151ba1ca8b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-kv62p"
Jul 23 14:57:00.121903 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:00.121864    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7df58b86-f921-4316-b8b1-f48137124a76-registry-tls\") pod \"image-registry-5cf48ff8c9-p9xlr\" (UID: \"7df58b86-f921-4316-b8b1-f48137124a76\") " pod="openshift-image-registry/image-registry-5cf48ff8c9-p9xlr"
Jul 23 14:57:00.124380 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:00.124351    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7df58b86-f921-4316-b8b1-f48137124a76-registry-tls\") pod \"image-registry-5cf48ff8c9-p9xlr\" (UID: \"7df58b86-f921-4316-b8b1-f48137124a76\") " pod="openshift-image-registry/image-registry-5cf48ff8c9-p9xlr"
Jul 23 14:57:00.132406 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:00.132382    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-kv62p"
Jul 23 14:57:00.241745 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:00.241714    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5cf48ff8c9-p9xlr"
Jul 23 14:57:00.255225 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:00.255197    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-kv62p"]
Jul 23 14:57:00.372433 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:00.372368    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5cf48ff8c9-p9xlr"]
Jul 23 14:57:00.375335 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:57:00.375304    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7df58b86_f921_4316_b8b1_f48137124a76.slice/crio-b7326f51cc138e2aafaeb77b583933a9876fff66ec7f1454c1fa9899f3d74bc1 WatchSource:0}: Error finding container b7326f51cc138e2aafaeb77b583933a9876fff66ec7f1454c1fa9899f3d74bc1: Status 404 returned error can't find the container with id b7326f51cc138e2aafaeb77b583933a9876fff66ec7f1454c1fa9899f3d74bc1
Jul 23 14:57:00.870948 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:00.870912    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-kv62p" event={"ID":"357d9f32-903d-472b-bf56-f5151ba1ca8b","Type":"ContainerStarted","Data":"aad022ad5ed54ffd1311adea416fb601ed181e581345e06001a040dc880fd1b1"}
Jul 23 14:57:00.872196 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:00.872162    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5cf48ff8c9-p9xlr" event={"ID":"7df58b86-f921-4316-b8b1-f48137124a76","Type":"ContainerStarted","Data":"9d1a6350541d81d6effeb58cd65c5da76043d8af9823280770ac8cdb92966cd3"}
Jul 23 14:57:00.872336 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:00.872211    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5cf48ff8c9-p9xlr" event={"ID":"7df58b86-f921-4316-b8b1-f48137124a76","Type":"ContainerStarted","Data":"b7326f51cc138e2aafaeb77b583933a9876fff66ec7f1454c1fa9899f3d74bc1"}
Jul 23 14:57:00.872336 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:00.872296    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5cf48ff8c9-p9xlr"
Jul 23 14:57:00.891912 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:00.891860    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5cf48ff8c9-p9xlr" podStartSLOduration=16.891842953 podStartE2EDuration="16.891842953s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 14:56:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 14:57:00.891721923 +0000 UTC m=+262.399595672" watchObservedRunningTime="2026-07-23 14:57:00.891842953 +0000 UTC m=+262.399716704"
Jul 23 14:57:02.880846 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:02.880813    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-kv62p" event={"ID":"357d9f32-903d-472b-bf56-f5151ba1ca8b","Type":"ContainerStarted","Data":"ee1c134776de958647ec66d0f0d48d68a33555012ce00d786e27b21d0a996c46"}
Jul 23 14:57:02.881218 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:02.880873    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-kv62p" event={"ID":"357d9f32-903d-472b-bf56-f5151ba1ca8b","Type":"ContainerStarted","Data":"e4f683f44cd6856d16a86fab99a0a7b4da6753d0f270858699c403686c52bd2a"}
Jul 23 14:57:02.900444 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:02.900386    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-kv62p" podStartSLOduration=16.902993752 podStartE2EDuration="18.900366701s" totalImagesPullingTime="1.997372949s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 14:56:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 14:57:02.897801509 +0000 UTC m=+264.405675271" watchObservedRunningTime="2026-07-23 14:57:02.900366701 +0000 UTC m=+264.408240449"
Jul 23 14:57:03.949356 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:03.949321    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/823379c6-9443-4acd-9c9f-8c300490b6b2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5xjtm\" (UID: \"823379c6-9443-4acd-9c9f-8c300490b6b2\") " pod="openshift-insights/insights-runtime-extractor-5xjtm"
Jul 23 14:57:03.949781 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:03.949375    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/823379c6-9443-4acd-9c9f-8c300490b6b2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5xjtm\" (UID: \"823379c6-9443-4acd-9c9f-8c300490b6b2\") " pod="openshift-insights/insights-runtime-extractor-5xjtm"
Jul 23 14:57:03.949836 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:03.949795    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/823379c6-9443-4acd-9c9f-8c300490b6b2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5xjtm\" (UID: \"823379c6-9443-4acd-9c9f-8c300490b6b2\") " pod="openshift-insights/insights-runtime-extractor-5xjtm"
Jul 23 14:57:03.951699 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:03.951680    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/823379c6-9443-4acd-9c9f-8c300490b6b2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5xjtm\" (UID: \"823379c6-9443-4acd-9c9f-8c300490b6b2\") " pod="openshift-insights/insights-runtime-extractor-5xjtm"
Jul 23 14:57:04.031314 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:04.031282    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-5xjtm"
Jul 23 14:57:04.157021 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:04.156991    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-5xjtm"]
Jul 23 14:57:04.160357 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:57:04.160332    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod823379c6_9443_4acd_9c9f_8c300490b6b2.slice/crio-167d754aefa86dac6a018dd140985f39bac568b147c297d2aae8606b1fe3a0c6 WatchSource:0}: Error finding container 167d754aefa86dac6a018dd140985f39bac568b147c297d2aae8606b1fe3a0c6: Status 404 returned error can't find the container with id 167d754aefa86dac6a018dd140985f39bac568b147c297d2aae8606b1fe3a0c6
Jul 23 14:57:04.888128 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:04.888091    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5xjtm" event={"ID":"823379c6-9443-4acd-9c9f-8c300490b6b2","Type":"ContainerStarted","Data":"167d754aefa86dac6a018dd140985f39bac568b147c297d2aae8606b1fe3a0c6"}
Jul 23 14:57:09.909719 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:09.909603    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5xjtm" event={"ID":"823379c6-9443-4acd-9c9f-8c300490b6b2","Type":"ContainerStarted","Data":"ce3c3e14b208c8d400b39f45c8f4ba9d854b271b87283d0a1ae8c6ccc290340b"}
Jul 23 14:57:12.921095 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:12.921064    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5xjtm" event={"ID":"823379c6-9443-4acd-9c9f-8c300490b6b2","Type":"ContainerStarted","Data":"266bb16736e1c8bf7bf14c1ddbcc5fa8351e2d4bc7161cbbba9b67dcedd6cc8b"}
Jul 23 14:57:13.682331 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:13.682101    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-5bc67dcd46-rzmqx"]
Jul 23 14:57:13.685452 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:13.685429    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-5bc67dcd46-rzmqx"
Jul 23 14:57:13.688293 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:13.688267    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-l8pb6\""
Jul 23 14:57:13.688428 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:13.688312    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 23 14:57:13.688608 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:13.688584    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 23 14:57:13.691951 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:13.691928    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5cf48ff8c9-p9xlr"]
Jul 23 14:57:13.700312 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:13.700285    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-5bc67dcd46-rzmqx"]
Jul 23 14:57:13.725466 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:13.725432    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htvdb\" (UniqueName: \"kubernetes.io/projected/ea3c4e5e-ccb4-417f-a9d9-5edc359ea988-kube-api-access-htvdb\") pod \"downloads-5bc67dcd46-rzmqx\" (UID: \"ea3c4e5e-ccb4-417f-a9d9-5edc359ea988\") " pod="openshift-console/downloads-5bc67dcd46-rzmqx"
Jul 23 14:57:13.826515 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:13.826482    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htvdb\" (UniqueName: \"kubernetes.io/projected/ea3c4e5e-ccb4-417f-a9d9-5edc359ea988-kube-api-access-htvdb\") pod \"downloads-5bc67dcd46-rzmqx\" (UID: \"ea3c4e5e-ccb4-417f-a9d9-5edc359ea988\") " pod="openshift-console/downloads-5bc67dcd46-rzmqx"
Jul 23 14:57:13.837818 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:13.837790    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-htvdb\" (UniqueName: \"kubernetes.io/projected/ea3c4e5e-ccb4-417f-a9d9-5edc359ea988-kube-api-access-htvdb\") pod \"downloads-5bc67dcd46-rzmqx\" (UID: \"ea3c4e5e-ccb4-417f-a9d9-5edc359ea988\") " pod="openshift-console/downloads-5bc67dcd46-rzmqx"
Jul 23 14:57:13.997856 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:13.997818    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-5bc67dcd46-rzmqx"
Jul 23 14:57:14.145583 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:14.145551    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-5bc67dcd46-rzmqx"]
Jul 23 14:57:14.150324 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:57:14.150289    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea3c4e5e_ccb4_417f_a9d9_5edc359ea988.slice/crio-09f7135dc0c66b55bb974a8913493e844f170f14f98a9101876dbc7d9e834246 WatchSource:0}: Error finding container 09f7135dc0c66b55bb974a8913493e844f170f14f98a9101876dbc7d9e834246: Status 404 returned error can't find the container with id 09f7135dc0c66b55bb974a8913493e844f170f14f98a9101876dbc7d9e834246
Jul 23 14:57:14.774860 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:14.774825    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-cbdfb4fbf-nl6bn"]
Jul 23 14:57:14.778372 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:14.778347    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cbdfb4fbf-nl6bn"
Jul 23 14:57:14.781215 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:14.781188    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 23 14:57:14.781458 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:14.781436    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 23 14:57:14.781570 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:14.781465    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 23 14:57:14.781964 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:14.781680    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-ld9fz\""
Jul 23 14:57:14.781964 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:14.781680    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 23 14:57:14.782715 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:14.782564    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 23 14:57:14.791719 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:14.791679    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-cbdfb4fbf-nl6bn"]
Jul 23 14:57:14.836273 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:14.836209    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6t2xz\" (UniqueName: \"kubernetes.io/projected/a37889b3-f376-4734-b5d4-812cefd9c0fc-kube-api-access-6t2xz\") pod \"console-cbdfb4fbf-nl6bn\" (UID: \"a37889b3-f376-4734-b5d4-812cefd9c0fc\") " pod="openshift-console/console-cbdfb4fbf-nl6bn"
Jul 23 14:57:14.836453 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:14.836311    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a37889b3-f376-4734-b5d4-812cefd9c0fc-console-serving-cert\") pod \"console-cbdfb4fbf-nl6bn\" (UID: \"a37889b3-f376-4734-b5d4-812cefd9c0fc\") " pod="openshift-console/console-cbdfb4fbf-nl6bn"
Jul 23 14:57:14.836453 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:14.836381    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a37889b3-f376-4734-b5d4-812cefd9c0fc-console-config\") pod \"console-cbdfb4fbf-nl6bn\" (UID: \"a37889b3-f376-4734-b5d4-812cefd9c0fc\") " pod="openshift-console/console-cbdfb4fbf-nl6bn"
Jul 23 14:57:14.836563 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:14.836473    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a37889b3-f376-4734-b5d4-812cefd9c0fc-oauth-serving-cert\") pod \"console-cbdfb4fbf-nl6bn\" (UID: \"a37889b3-f376-4734-b5d4-812cefd9c0fc\") " pod="openshift-console/console-cbdfb4fbf-nl6bn"
Jul 23 14:57:14.836563 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:14.836528    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a37889b3-f376-4734-b5d4-812cefd9c0fc-service-ca\") pod \"console-cbdfb4fbf-nl6bn\" (UID: \"a37889b3-f376-4734-b5d4-812cefd9c0fc\") " pod="openshift-console/console-cbdfb4fbf-nl6bn"
Jul 23 14:57:14.836668 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:14.836579    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a37889b3-f376-4734-b5d4-812cefd9c0fc-console-oauth-config\") pod \"console-cbdfb4fbf-nl6bn\" (UID: \"a37889b3-f376-4734-b5d4-812cefd9c0fc\") " pod="openshift-console/console-cbdfb4fbf-nl6bn"
Jul 23 14:57:14.931268 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:14.931205    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-5bc67dcd46-rzmqx" event={"ID":"ea3c4e5e-ccb4-417f-a9d9-5edc359ea988","Type":"ContainerStarted","Data":"09f7135dc0c66b55bb974a8913493e844f170f14f98a9101876dbc7d9e834246"}
Jul 23 14:57:14.937639 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:14.937604    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a37889b3-f376-4734-b5d4-812cefd9c0fc-oauth-serving-cert\") pod \"console-cbdfb4fbf-nl6bn\" (UID: \"a37889b3-f376-4734-b5d4-812cefd9c0fc\") " pod="openshift-console/console-cbdfb4fbf-nl6bn"
Jul 23 14:57:14.937795 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:14.937658    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a37889b3-f376-4734-b5d4-812cefd9c0fc-service-ca\") pod \"console-cbdfb4fbf-nl6bn\" (UID: \"a37889b3-f376-4734-b5d4-812cefd9c0fc\") " pod="openshift-console/console-cbdfb4fbf-nl6bn"
Jul 23 14:57:14.937795 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:14.937705    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a37889b3-f376-4734-b5d4-812cefd9c0fc-console-oauth-config\") pod \"console-cbdfb4fbf-nl6bn\" (UID: \"a37889b3-f376-4734-b5d4-812cefd9c0fc\") " pod="openshift-console/console-cbdfb4fbf-nl6bn"
Jul 23 14:57:14.937795 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:14.937735    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6t2xz\" (UniqueName: \"kubernetes.io/projected/a37889b3-f376-4734-b5d4-812cefd9c0fc-kube-api-access-6t2xz\") pod \"console-cbdfb4fbf-nl6bn\" (UID: \"a37889b3-f376-4734-b5d4-812cefd9c0fc\") " pod="openshift-console/console-cbdfb4fbf-nl6bn"
Jul 23 14:57:14.937795 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:14.937770    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a37889b3-f376-4734-b5d4-812cefd9c0fc-console-serving-cert\") pod \"console-cbdfb4fbf-nl6bn\" (UID: \"a37889b3-f376-4734-b5d4-812cefd9c0fc\") " pod="openshift-console/console-cbdfb4fbf-nl6bn"
Jul 23 14:57:14.938014 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:14.937817    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a37889b3-f376-4734-b5d4-812cefd9c0fc-console-config\") pod \"console-cbdfb4fbf-nl6bn\" (UID: \"a37889b3-f376-4734-b5d4-812cefd9c0fc\") " pod="openshift-console/console-cbdfb4fbf-nl6bn"
Jul 23 14:57:14.938666 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:14.938541    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a37889b3-f376-4734-b5d4-812cefd9c0fc-console-config\") pod \"console-cbdfb4fbf-nl6bn\" (UID: \"a37889b3-f376-4734-b5d4-812cefd9c0fc\") " pod="openshift-console/console-cbdfb4fbf-nl6bn"
Jul 23 14:57:14.938666 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:14.938539    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a37889b3-f376-4734-b5d4-812cefd9c0fc-oauth-serving-cert\") pod \"console-cbdfb4fbf-nl6bn\" (UID: \"a37889b3-f376-4734-b5d4-812cefd9c0fc\") " pod="openshift-console/console-cbdfb4fbf-nl6bn"
Jul 23 14:57:14.938858 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:14.938773    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a37889b3-f376-4734-b5d4-812cefd9c0fc-service-ca\") pod \"console-cbdfb4fbf-nl6bn\" (UID: \"a37889b3-f376-4734-b5d4-812cefd9c0fc\") " pod="openshift-console/console-cbdfb4fbf-nl6bn"
Jul 23 14:57:14.941479 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:14.941454    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a37889b3-f376-4734-b5d4-812cefd9c0fc-console-serving-cert\") pod \"console-cbdfb4fbf-nl6bn\" (UID: \"a37889b3-f376-4734-b5d4-812cefd9c0fc\") " pod="openshift-console/console-cbdfb4fbf-nl6bn"
Jul 23 14:57:14.941614 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:14.941487    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a37889b3-f376-4734-b5d4-812cefd9c0fc-console-oauth-config\") pod \"console-cbdfb4fbf-nl6bn\" (UID: \"a37889b3-f376-4734-b5d4-812cefd9c0fc\") " pod="openshift-console/console-cbdfb4fbf-nl6bn"
Jul 23 14:57:14.946663 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:14.946635    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6t2xz\" (UniqueName: \"kubernetes.io/projected/a37889b3-f376-4734-b5d4-812cefd9c0fc-kube-api-access-6t2xz\") pod \"console-cbdfb4fbf-nl6bn\" (UID: \"a37889b3-f376-4734-b5d4-812cefd9c0fc\") " pod="openshift-console/console-cbdfb4fbf-nl6bn"
Jul 23 14:57:15.091202 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:15.091116    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cbdfb4fbf-nl6bn"
Jul 23 14:57:16.677796 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:16.677767    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-cbdfb4fbf-nl6bn"]
Jul 23 14:57:16.680719 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:57:16.680681    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda37889b3_f376_4734_b5d4_812cefd9c0fc.slice/crio-52b3b1943e9c67d37f46f4929ef18aeb3a622ab770f378a6190921fad1e558e7 WatchSource:0}: Error finding container 52b3b1943e9c67d37f46f4929ef18aeb3a622ab770f378a6190921fad1e558e7: Status 404 returned error can't find the container with id 52b3b1943e9c67d37f46f4929ef18aeb3a622ab770f378a6190921fad1e558e7
Jul 23 14:57:16.941043 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:16.940924    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5xjtm" event={"ID":"823379c6-9443-4acd-9c9f-8c300490b6b2","Type":"ContainerStarted","Data":"052ab617865859ce7758f406b91106d19a812ed8cfc2b5fd48b3637c596f8f13"}
Jul 23 14:57:16.942331 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:16.942291    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cbdfb4fbf-nl6bn" event={"ID":"a37889b3-f376-4734-b5d4-812cefd9c0fc","Type":"ContainerStarted","Data":"52b3b1943e9c67d37f46f4929ef18aeb3a622ab770f378a6190921fad1e558e7"}
Jul 23 14:57:16.962092 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:16.962041    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-5xjtm" podStartSLOduration=16.635520812 podStartE2EDuration="28.962021897s" totalImagesPullingTime="12.326501085s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 14:56:48 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 14:57:16.95993298 +0000 UTC m=+278.467806729" watchObservedRunningTime="2026-07-23 14:57:16.962021897 +0000 UTC m=+278.469895646"
Jul 23 14:57:19.553307 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:57:19.553259    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-scspg" podUID="a2a2b51d-3d22-42dd-96b6-093bdf7c3f7a"
Jul 23 14:57:19.960473 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:19.960420    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cbdfb4fbf-nl6bn" event={"ID":"a37889b3-f376-4734-b5d4-812cefd9c0fc","Type":"ContainerStarted","Data":"109029d5789480bb59189c57a1512683251e8dfe61c943141d8aab45c895713a"}
Jul 23 14:57:19.960473 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:19.960458    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-scspg"
Jul 23 14:57:19.980508 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:19.980446    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-cbdfb4fbf-nl6bn" podStartSLOduration=2.8772009179999998 podStartE2EDuration="5.980429653s" totalImagesPullingTime="3.103228735s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 14:57:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 14:57:19.978891276 +0000 UTC m=+281.486765023" watchObservedRunningTime="2026-07-23 14:57:19.980429653 +0000 UTC m=+281.488303401"
Jul 23 14:57:23.411100 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:23.411053    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a2a2b51d-3d22-42dd-96b6-093bdf7c3f7a-metrics-tls\") pod \"dns-default-scspg\" (UID: \"a2a2b51d-3d22-42dd-96b6-093bdf7c3f7a\") " pod="openshift-dns/dns-default-scspg"
Jul 23 14:57:23.413793 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:23.413760    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a2a2b51d-3d22-42dd-96b6-093bdf7c3f7a-metrics-tls\") pod \"dns-default-scspg\" (UID: \"a2a2b51d-3d22-42dd-96b6-093bdf7c3f7a\") " pod="openshift-dns/dns-default-scspg"
Jul 23 14:57:23.563919 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:23.563881    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-zzbs4\""
Jul 23 14:57:23.571278 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:23.571243    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-scspg"
Jul 23 14:57:23.698763 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:23.698724    2560 patch_prober.go:28] interesting pod/image-registry-5cf48ff8c9-p9xlr container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 23 14:57:23.698971 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:23.698790    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-5cf48ff8c9-p9xlr" podUID="7df58b86-f921-4316-b8b1-f48137124a76" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 23 14:57:23.727136 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:23.727102    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-scspg"]
Jul 23 14:57:23.730879 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:57:23.730841    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2a2b51d_3d22_42dd_96b6_093bdf7c3f7a.slice/crio-4dc25a1535dde0ca0a73f6a4191a5252d34e02aecb25fdd975500a2d2797c413 WatchSource:0}: Error finding container 4dc25a1535dde0ca0a73f6a4191a5252d34e02aecb25fdd975500a2d2797c413: Status 404 returned error can't find the container with id 4dc25a1535dde0ca0a73f6a4191a5252d34e02aecb25fdd975500a2d2797c413
Jul 23 14:57:23.975458 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:23.975425    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-scspg" event={"ID":"a2a2b51d-3d22-42dd-96b6-093bdf7c3f7a","Type":"ContainerStarted","Data":"4dc25a1535dde0ca0a73f6a4191a5252d34e02aecb25fdd975500a2d2797c413"}
Jul 23 14:57:25.091433 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:25.091395    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-cbdfb4fbf-nl6bn"
Jul 23 14:57:25.091919 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:25.091470    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-cbdfb4fbf-nl6bn"
Jul 23 14:57:25.093108 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:25.093075    2560 patch_prober.go:28] interesting pod/console-cbdfb4fbf-nl6bn container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.134.0.16:8443/health\": dial tcp 10.134.0.16:8443: connect: connection refused" start-of-body=
Jul 23 14:57:25.093259 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:25.093133    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-cbdfb4fbf-nl6bn" podUID="a37889b3-f376-4734-b5d4-812cefd9c0fc" containerName="console" probeResult="failure" output="Get \"https://10.134.0.16:8443/health\": dial tcp 10.134.0.16:8443: connect: connection refused"
Jul 23 14:57:25.546092 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:25.546053    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-wt7xl"]
Jul 23 14:57:25.550690 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:25.550662    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-wt7xl"
Jul 23 14:57:25.554886 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:25.554859    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 23 14:57:25.555022 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:25.554915    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 23 14:57:25.555022 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:25.554859    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 23 14:57:25.555214 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:25.555193    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 23 14:57:25.555695 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:25.555666    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 23 14:57:25.556039 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:25.556013    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 23 14:57:25.556139 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:25.556022    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-62kkx\""
Jul 23 14:57:25.633264 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:25.633212    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvvk8\" (UniqueName: \"kubernetes.io/projected/a2e2585d-0204-4faa-807e-b1e28263c9bf-kube-api-access-rvvk8\") pod \"node-exporter-wt7xl\" (UID: \"a2e2585d-0204-4faa-807e-b1e28263c9bf\") " pod="openshift-monitoring/node-exporter-wt7xl"
Jul 23 14:57:25.633455 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:25.633287    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a2e2585d-0204-4faa-807e-b1e28263c9bf-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wt7xl\" (UID: \"a2e2585d-0204-4faa-807e-b1e28263c9bf\") " pod="openshift-monitoring/node-exporter-wt7xl"
Jul 23 14:57:25.633455 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:25.633323    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a2e2585d-0204-4faa-807e-b1e28263c9bf-node-exporter-wtmp\") pod \"node-exporter-wt7xl\" (UID: \"a2e2585d-0204-4faa-807e-b1e28263c9bf\") " pod="openshift-monitoring/node-exporter-wt7xl"
Jul 23 14:57:25.633455 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:25.633354    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a2e2585d-0204-4faa-807e-b1e28263c9bf-node-exporter-accelerators-collector-config\") pod \"node-exporter-wt7xl\" (UID: \"a2e2585d-0204-4faa-807e-b1e28263c9bf\") " pod="openshift-monitoring/node-exporter-wt7xl"
Jul 23 14:57:25.633455 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:25.633403    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a2e2585d-0204-4faa-807e-b1e28263c9bf-sys\") pod \"node-exporter-wt7xl\" (UID: \"a2e2585d-0204-4faa-807e-b1e28263c9bf\") " pod="openshift-monitoring/node-exporter-wt7xl"
Jul 23 14:57:25.633455 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:25.633438    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a2e2585d-0204-4faa-807e-b1e28263c9bf-node-exporter-textfile\") pod \"node-exporter-wt7xl\" (UID: \"a2e2585d-0204-4faa-807e-b1e28263c9bf\") " pod="openshift-monitoring/node-exporter-wt7xl"
Jul 23 14:57:25.633700 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:25.633506    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a2e2585d-0204-4faa-807e-b1e28263c9bf-root\") pod \"node-exporter-wt7xl\" (UID: \"a2e2585d-0204-4faa-807e-b1e28263c9bf\") " pod="openshift-monitoring/node-exporter-wt7xl"
Jul 23 14:57:25.633700 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:25.633545    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a2e2585d-0204-4faa-807e-b1e28263c9bf-metrics-client-ca\") pod \"node-exporter-wt7xl\" (UID: \"a2e2585d-0204-4faa-807e-b1e28263c9bf\") " pod="openshift-monitoring/node-exporter-wt7xl"
Jul 23 14:57:25.633700 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:25.633600    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a2e2585d-0204-4faa-807e-b1e28263c9bf-node-exporter-tls\") pod \"node-exporter-wt7xl\" (UID: \"a2e2585d-0204-4faa-807e-b1e28263c9bf\") " pod="openshift-monitoring/node-exporter-wt7xl"
Jul 23 14:57:25.734749 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:25.734710    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a2e2585d-0204-4faa-807e-b1e28263c9bf-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wt7xl\" (UID: \"a2e2585d-0204-4faa-807e-b1e28263c9bf\") " pod="openshift-monitoring/node-exporter-wt7xl"
Jul 23 14:57:25.734950 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:25.734764    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a2e2585d-0204-4faa-807e-b1e28263c9bf-node-exporter-wtmp\") pod \"node-exporter-wt7xl\" (UID: \"a2e2585d-0204-4faa-807e-b1e28263c9bf\") " pod="openshift-monitoring/node-exporter-wt7xl"
Jul 23 14:57:25.734950 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:25.734797    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a2e2585d-0204-4faa-807e-b1e28263c9bf-node-exporter-accelerators-collector-config\") pod \"node-exporter-wt7xl\" (UID: \"a2e2585d-0204-4faa-807e-b1e28263c9bf\") " pod="openshift-monitoring/node-exporter-wt7xl"
Jul 23 14:57:25.734950 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:25.734826    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a2e2585d-0204-4faa-807e-b1e28263c9bf-sys\") pod \"node-exporter-wt7xl\" (UID: \"a2e2585d-0204-4faa-807e-b1e28263c9bf\") " pod="openshift-monitoring/node-exporter-wt7xl"
Jul 23 14:57:25.734950 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:25.734852    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a2e2585d-0204-4faa-807e-b1e28263c9bf-node-exporter-textfile\") pod \"node-exporter-wt7xl\" (UID: \"a2e2585d-0204-4faa-807e-b1e28263c9bf\") " pod="openshift-monitoring/node-exporter-wt7xl"
Jul 23 14:57:25.734950 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:25.734881    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a2e2585d-0204-4faa-807e-b1e28263c9bf-root\") pod \"node-exporter-wt7xl\" (UID: \"a2e2585d-0204-4faa-807e-b1e28263c9bf\") " pod="openshift-monitoring/node-exporter-wt7xl"
Jul 23 14:57:25.734950 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:25.734909    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a2e2585d-0204-4faa-807e-b1e28263c9bf-metrics-client-ca\") pod \"node-exporter-wt7xl\" (UID: \"a2e2585d-0204-4faa-807e-b1e28263c9bf\") " pod="openshift-monitoring/node-exporter-wt7xl"
Jul 23 14:57:25.734950 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:25.734921    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a2e2585d-0204-4faa-807e-b1e28263c9bf-sys\") pod \"node-exporter-wt7xl\" (UID: \"a2e2585d-0204-4faa-807e-b1e28263c9bf\") " pod="openshift-monitoring/node-exporter-wt7xl"
Jul 23 14:57:25.735380 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:25.734958    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a2e2585d-0204-4faa-807e-b1e28263c9bf-node-exporter-tls\") pod \"node-exporter-wt7xl\" (UID: \"a2e2585d-0204-4faa-807e-b1e28263c9bf\") " pod="openshift-monitoring/node-exporter-wt7xl"
Jul 23 14:57:25.735380 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:25.734982    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a2e2585d-0204-4faa-807e-b1e28263c9bf-node-exporter-wtmp\") pod \"node-exporter-wt7xl\" (UID: \"a2e2585d-0204-4faa-807e-b1e28263c9bf\") " pod="openshift-monitoring/node-exporter-wt7xl"
Jul 23 14:57:25.735380 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:25.734990    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a2e2585d-0204-4faa-807e-b1e28263c9bf-root\") pod \"node-exporter-wt7xl\" (UID: \"a2e2585d-0204-4faa-807e-b1e28263c9bf\") " pod="openshift-monitoring/node-exporter-wt7xl"
Jul 23 14:57:25.735380 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:25.735015    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvvk8\" (UniqueName: \"kubernetes.io/projected/a2e2585d-0204-4faa-807e-b1e28263c9bf-kube-api-access-rvvk8\") pod \"node-exporter-wt7xl\" (UID: \"a2e2585d-0204-4faa-807e-b1e28263c9bf\") " pod="openshift-monitoring/node-exporter-wt7xl"
Jul 23 14:57:25.735552 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:25.735393    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a2e2585d-0204-4faa-807e-b1e28263c9bf-node-exporter-textfile\") pod \"node-exporter-wt7xl\" (UID: \"a2e2585d-0204-4faa-807e-b1e28263c9bf\") " pod="openshift-monitoring/node-exporter-wt7xl"
Jul 23 14:57:25.735628 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:25.735611    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a2e2585d-0204-4faa-807e-b1e28263c9bf-node-exporter-accelerators-collector-config\") pod \"node-exporter-wt7xl\" (UID: \"a2e2585d-0204-4faa-807e-b1e28263c9bf\") " pod="openshift-monitoring/node-exporter-wt7xl"
Jul 23 14:57:25.735705 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:25.735685    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a2e2585d-0204-4faa-807e-b1e28263c9bf-metrics-client-ca\") pod \"node-exporter-wt7xl\" (UID: \"a2e2585d-0204-4faa-807e-b1e28263c9bf\") " pod="openshift-monitoring/node-exporter-wt7xl"
Jul 23 14:57:25.737899 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:25.737880    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a2e2585d-0204-4faa-807e-b1e28263c9bf-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wt7xl\" (UID: \"a2e2585d-0204-4faa-807e-b1e28263c9bf\") " pod="openshift-monitoring/node-exporter-wt7xl"
Jul 23 14:57:25.738044 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:25.738007    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a2e2585d-0204-4faa-807e-b1e28263c9bf-node-exporter-tls\") pod \"node-exporter-wt7xl\" (UID: \"a2e2585d-0204-4faa-807e-b1e28263c9bf\") " pod="openshift-monitoring/node-exporter-wt7xl"
Jul 23 14:57:25.745047 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:25.745017    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvvk8\" (UniqueName: \"kubernetes.io/projected/a2e2585d-0204-4faa-807e-b1e28263c9bf-kube-api-access-rvvk8\") pod \"node-exporter-wt7xl\" (UID: \"a2e2585d-0204-4faa-807e-b1e28263c9bf\") " pod="openshift-monitoring/node-exporter-wt7xl"
Jul 23 14:57:25.863482 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:25.863398    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-wt7xl"
Jul 23 14:57:26.602452 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:26.602418    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 23 14:57:26.606186 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:26.606163    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 23 14:57:26.609047 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:26.609021    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 23 14:57:26.609189 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:26.609021    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 23 14:57:26.609324 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:26.609310    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 23 14:57:26.609389 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:26.609338    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 23 14:57:26.609449 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:26.609387    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 23 14:57:26.609449 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:26.609433    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 23 14:57:26.609575 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:26.609560    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-6nz66\""
Jul 23 14:57:26.610476 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:26.610459    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 23 14:57:26.610476 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:26.610462    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 23 14:57:26.610638 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:26.610565    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 23 14:57:26.619276 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:26.619219    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 23 14:57:26.749586 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:26.749545    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/3e1cce8d-f0cf-4f0e-ae15-1365159a053a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"3e1cce8d-f0cf-4f0e-ae15-1365159a053a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 14:57:26.749780 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:26.749599    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3e1cce8d-f0cf-4f0e-ae15-1365159a053a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"3e1cce8d-f0cf-4f0e-ae15-1365159a053a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 14:57:26.749780 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:26.749632    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3e1cce8d-f0cf-4f0e-ae15-1365159a053a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"3e1cce8d-f0cf-4f0e-ae15-1365159a053a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 14:57:26.749780 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:26.749661    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3e1cce8d-f0cf-4f0e-ae15-1365159a053a-web-config\") pod \"alertmanager-main-0\" (UID: \"3e1cce8d-f0cf-4f0e-ae15-1365159a053a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 14:57:26.749780 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:26.749690    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3e1cce8d-f0cf-4f0e-ae15-1365159a053a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"3e1cce8d-f0cf-4f0e-ae15-1365159a053a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 14:57:26.749780 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:26.749771    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3e1cce8d-f0cf-4f0e-ae15-1365159a053a-config-volume\") pod \"alertmanager-main-0\" (UID: \"3e1cce8d-f0cf-4f0e-ae15-1365159a053a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 14:57:26.750109 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:26.749832    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3e1cce8d-f0cf-4f0e-ae15-1365159a053a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"3e1cce8d-f0cf-4f0e-ae15-1365159a053a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 14:57:26.750109 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:26.749860    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3e1cce8d-f0cf-4f0e-ae15-1365159a053a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"3e1cce8d-f0cf-4f0e-ae15-1365159a053a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 14:57:26.750109 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:26.749888    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/3e1cce8d-f0cf-4f0e-ae15-1365159a053a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"3e1cce8d-f0cf-4f0e-ae15-1365159a053a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 14:57:26.750109 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:26.749912    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbs9x\" (UniqueName: \"kubernetes.io/projected/3e1cce8d-f0cf-4f0e-ae15-1365159a053a-kube-api-access-qbs9x\") pod \"alertmanager-main-0\" (UID: \"3e1cce8d-f0cf-4f0e-ae15-1365159a053a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 14:57:26.750109 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:26.749938    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3e1cce8d-f0cf-4f0e-ae15-1365159a053a-config-out\") pod \"alertmanager-main-0\" (UID: \"3e1cce8d-f0cf-4f0e-ae15-1365159a053a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 14:57:26.750109 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:26.749977    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3e1cce8d-f0cf-4f0e-ae15-1365159a053a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"3e1cce8d-f0cf-4f0e-ae15-1365159a053a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 14:57:26.750109 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:26.750010    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3e1cce8d-f0cf-4f0e-ae15-1365159a053a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"3e1cce8d-f0cf-4f0e-ae15-1365159a053a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 14:57:26.851135 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:26.851094    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3e1cce8d-f0cf-4f0e-ae15-1365159a053a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"3e1cce8d-f0cf-4f0e-ae15-1365159a053a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 14:57:26.851361 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:26.851143    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3e1cce8d-f0cf-4f0e-ae15-1365159a053a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"3e1cce8d-f0cf-4f0e-ae15-1365159a053a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 14:57:26.851361 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:26.851171    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/3e1cce8d-f0cf-4f0e-ae15-1365159a053a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"3e1cce8d-f0cf-4f0e-ae15-1365159a053a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 14:57:26.851361 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:26.851246    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbs9x\" (UniqueName: \"kubernetes.io/projected/3e1cce8d-f0cf-4f0e-ae15-1365159a053a-kube-api-access-qbs9x\") pod \"alertmanager-main-0\" (UID: \"3e1cce8d-f0cf-4f0e-ae15-1365159a053a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 14:57:26.851361 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:26.851300    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3e1cce8d-f0cf-4f0e-ae15-1365159a053a-config-out\") pod \"alertmanager-main-0\" (UID: \"3e1cce8d-f0cf-4f0e-ae15-1365159a053a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 14:57:26.851361 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:26.851350    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3e1cce8d-f0cf-4f0e-ae15-1365159a053a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"3e1cce8d-f0cf-4f0e-ae15-1365159a053a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 14:57:26.851657 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:26.851389    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3e1cce8d-f0cf-4f0e-ae15-1365159a053a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"3e1cce8d-f0cf-4f0e-ae15-1365159a053a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 14:57:26.851657 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:26.851444    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/3e1cce8d-f0cf-4f0e-ae15-1365159a053a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"3e1cce8d-f0cf-4f0e-ae15-1365159a053a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 14:57:26.851657 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:26.851469    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3e1cce8d-f0cf-4f0e-ae15-1365159a053a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"3e1cce8d-f0cf-4f0e-ae15-1365159a053a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 14:57:26.851657 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:26.851487    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3e1cce8d-f0cf-4f0e-ae15-1365159a053a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"3e1cce8d-f0cf-4f0e-ae15-1365159a053a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 14:57:26.851657 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:26.851503    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3e1cce8d-f0cf-4f0e-ae15-1365159a053a-web-config\") pod \"alertmanager-main-0\" (UID: \"3e1cce8d-f0cf-4f0e-ae15-1365159a053a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 14:57:26.851657 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:26.851528    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3e1cce8d-f0cf-4f0e-ae15-1365159a053a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"3e1cce8d-f0cf-4f0e-ae15-1365159a053a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 14:57:26.851657 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:26.851591    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3e1cce8d-f0cf-4f0e-ae15-1365159a053a-config-volume\") pod \"alertmanager-main-0\" (UID: \"3e1cce8d-f0cf-4f0e-ae15-1365159a053a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 14:57:26.851657 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:26.851593    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/3e1cce8d-f0cf-4f0e-ae15-1365159a053a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"3e1cce8d-f0cf-4f0e-ae15-1365159a053a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 14:57:26.852061 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:26.852029    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3e1cce8d-f0cf-4f0e-ae15-1365159a053a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"3e1cce8d-f0cf-4f0e-ae15-1365159a053a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 14:57:26.853025 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:26.852963    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3e1cce8d-f0cf-4f0e-ae15-1365159a053a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"3e1cce8d-f0cf-4f0e-ae15-1365159a053a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 14:57:26.854838 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:26.854805    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3e1cce8d-f0cf-4f0e-ae15-1365159a053a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"3e1cce8d-f0cf-4f0e-ae15-1365159a053a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 14:57:26.855358 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:26.855288    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3e1cce8d-f0cf-4f0e-ae15-1365159a053a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"3e1cce8d-f0cf-4f0e-ae15-1365159a053a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 14:57:26.855358 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:26.855314    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3e1cce8d-f0cf-4f0e-ae15-1365159a053a-config-volume\") pod \"alertmanager-main-0\" (UID: \"3e1cce8d-f0cf-4f0e-ae15-1365159a053a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 14:57:26.855550 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:26.855521    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3e1cce8d-f0cf-4f0e-ae15-1365159a053a-config-out\") pod \"alertmanager-main-0\" (UID: \"3e1cce8d-f0cf-4f0e-ae15-1365159a053a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 14:57:26.856048 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:26.856007    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/3e1cce8d-f0cf-4f0e-ae15-1365159a053a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"3e1cce8d-f0cf-4f0e-ae15-1365159a053a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 14:57:26.856313 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:26.856267    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3e1cce8d-f0cf-4f0e-ae15-1365159a053a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"3e1cce8d-f0cf-4f0e-ae15-1365159a053a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 14:57:26.856420 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:26.856357    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3e1cce8d-f0cf-4f0e-ae15-1365159a053a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"3e1cce8d-f0cf-4f0e-ae15-1365159a053a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 14:57:26.857603 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:26.857559    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3e1cce8d-f0cf-4f0e-ae15-1365159a053a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"3e1cce8d-f0cf-4f0e-ae15-1365159a053a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 14:57:26.857714 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:26.857646    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3e1cce8d-f0cf-4f0e-ae15-1365159a053a-web-config\") pod \"alertmanager-main-0\" (UID: \"3e1cce8d-f0cf-4f0e-ae15-1365159a053a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 14:57:26.859879 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:26.859859    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbs9x\" (UniqueName: \"kubernetes.io/projected/3e1cce8d-f0cf-4f0e-ae15-1365159a053a-kube-api-access-qbs9x\") pod \"alertmanager-main-0\" (UID: \"3e1cce8d-f0cf-4f0e-ae15-1365159a053a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 14:57:26.918519 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:26.918479    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 23 14:57:30.303443 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:30.303406    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-29hwq"]
Jul 23 14:57:30.350799 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:30.350755    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-29hwq"]
Jul 23 14:57:30.350989 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:30.350863    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-29hwq"
Jul 23 14:57:30.354854 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:30.354827    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Jul 23 14:57:30.355006 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:30.354827    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-j2h7v\""
Jul 23 14:57:30.486878 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:30.486844    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e1ee8253-0c66-4e49-827b-42d03d36c796-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-29hwq\" (UID: \"e1ee8253-0c66-4e49-827b-42d03d36c796\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-29hwq"
Jul 23 14:57:30.486878 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:30.486881    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8fdb\" (UniqueName: \"kubernetes.io/projected/e1ee8253-0c66-4e49-827b-42d03d36c796-kube-api-access-g8fdb\") pod \"monitoring-plugin-7b7b7bc8bf-29hwq\" (UID: \"e1ee8253-0c66-4e49-827b-42d03d36c796\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-29hwq"
Jul 23 14:57:30.588398 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:30.588328    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e1ee8253-0c66-4e49-827b-42d03d36c796-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-29hwq\" (UID: \"e1ee8253-0c66-4e49-827b-42d03d36c796\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-29hwq"
Jul 23 14:57:30.588398 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:30.588363    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8fdb\" (UniqueName: \"kubernetes.io/projected/e1ee8253-0c66-4e49-827b-42d03d36c796-kube-api-access-g8fdb\") pod \"monitoring-plugin-7b7b7bc8bf-29hwq\" (UID: \"e1ee8253-0c66-4e49-827b-42d03d36c796\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-29hwq"
Jul 23 14:57:30.591330 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:30.591302    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e1ee8253-0c66-4e49-827b-42d03d36c796-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-29hwq\" (UID: \"e1ee8253-0c66-4e49-827b-42d03d36c796\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-29hwq"
Jul 23 14:57:30.598206 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:30.598161    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8fdb\" (UniqueName: \"kubernetes.io/projected/e1ee8253-0c66-4e49-827b-42d03d36c796-kube-api-access-g8fdb\") pod \"monitoring-plugin-7b7b7bc8bf-29hwq\" (UID: \"e1ee8253-0c66-4e49-827b-42d03d36c796\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-29hwq"
Jul 23 14:57:30.661986 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:30.661953    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-29hwq"
Jul 23 14:57:30.977338 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:30.977306    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-cbdfb4fbf-nl6bn"]
Jul 23 14:57:30.988599 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:57:30.988532    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2e2585d_0204_4faa_807e_b1e28263c9bf.slice/crio-aeec7f3c227476a04365fb10f72a597115d19bd54a6027dc00bc01dfa6bc4bd0 WatchSource:0}: Error finding container aeec7f3c227476a04365fb10f72a597115d19bd54a6027dc00bc01dfa6bc4bd0: Status 404 returned error can't find the container with id aeec7f3c227476a04365fb10f72a597115d19bd54a6027dc00bc01dfa6bc4bd0
Jul 23 14:57:31.004669 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:31.004631    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wt7xl" event={"ID":"a2e2585d-0204-4faa-807e-b1e28263c9bf","Type":"ContainerStarted","Data":"aeec7f3c227476a04365fb10f72a597115d19bd54a6027dc00bc01dfa6bc4bd0"}
Jul 23 14:57:31.137488 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:31.137451    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 23 14:57:31.149556 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:31.149525    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-29hwq"]
Jul 23 14:57:31.255479 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:57:31.255395    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e1cce8d_f0cf_4f0e_ae15_1365159a053a.slice/crio-bc7906f2c8012f2038f22e05049d9de909599e278251977230501c7f660bd16e WatchSource:0}: Error finding container bc7906f2c8012f2038f22e05049d9de909599e278251977230501c7f660bd16e: Status 404 returned error can't find the container with id bc7906f2c8012f2038f22e05049d9de909599e278251977230501c7f660bd16e
Jul 23 14:57:31.256007 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:57:31.255979    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1ee8253_0c66_4e49_827b_42d03d36c796.slice/crio-abe1deb6195b3399e410d1d1bb5e059e53e81a4148858320479cc20cf9089ad5 WatchSource:0}: Error finding container abe1deb6195b3399e410d1d1bb5e059e53e81a4148858320479cc20cf9089ad5: Status 404 returned error can't find the container with id abe1deb6195b3399e410d1d1bb5e059e53e81a4148858320479cc20cf9089ad5
Jul 23 14:57:31.714319 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:31.713171    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 23 14:57:31.718868 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:31.717741    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 14:57:31.724211 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:31.722975    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 23 14:57:31.724211 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:31.723697    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 23 14:57:31.724211 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:31.723921    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 23 14:57:31.724211 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:31.724093    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 23 14:57:31.725927 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:31.724555    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 23 14:57:31.725927 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:31.724979    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 23 14:57:31.729279 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:31.726351    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 23 14:57:31.729279 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:31.726602    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 23 14:57:31.729279 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:31.726759    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 23 14:57:31.729279 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:31.726899    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-qfqg2\""
Jul 23 14:57:31.729279 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:31.727965    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 23 14:57:31.729279 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:31.728196    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 23 14:57:31.729279 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:31.728392    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-6rr9scg9k0sk7\""
Jul 23 14:57:31.729279 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:31.728566    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 23 14:57:31.729279 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:31.728734    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 23 14:57:31.731454 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:31.730150    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 23 14:57:31.732775 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:31.732565    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 23 14:57:31.736246 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:31.736143    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 23 14:57:31.801086 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:31.801056    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0e02c795-e710-498b-90e9-7a846f16504b-web-config\") pod \"prometheus-k8s-0\" (UID: \"0e02c795-e710-498b-90e9-7a846f16504b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 14:57:31.801291 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:31.801097    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0e02c795-e710-498b-90e9-7a846f16504b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0e02c795-e710-498b-90e9-7a846f16504b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 14:57:31.801291 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:31.801126    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e02c795-e710-498b-90e9-7a846f16504b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0e02c795-e710-498b-90e9-7a846f16504b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 14:57:31.801291 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:31.801146    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0e02c795-e710-498b-90e9-7a846f16504b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0e02c795-e710-498b-90e9-7a846f16504b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 14:57:31.801291 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:31.801170    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e02c795-e710-498b-90e9-7a846f16504b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0e02c795-e710-498b-90e9-7a846f16504b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 14:57:31.801291 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:31.801191    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0e02c795-e710-498b-90e9-7a846f16504b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0e02c795-e710-498b-90e9-7a846f16504b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 14:57:31.801291 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:31.801259    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/0e02c795-e710-498b-90e9-7a846f16504b-config\") pod \"prometheus-k8s-0\" (UID: \"0e02c795-e710-498b-90e9-7a846f16504b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 14:57:31.801616 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:31.801298    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0e02c795-e710-498b-90e9-7a846f16504b-config-out\") pod \"prometheus-k8s-0\" (UID: \"0e02c795-e710-498b-90e9-7a846f16504b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 14:57:31.801616 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:31.801321    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e02c795-e710-498b-90e9-7a846f16504b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0e02c795-e710-498b-90e9-7a846f16504b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 14:57:31.801616 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:31.801342    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f68f6\" (UniqueName: \"kubernetes.io/projected/0e02c795-e710-498b-90e9-7a846f16504b-kube-api-access-f68f6\") pod \"prometheus-k8s-0\" (UID: \"0e02c795-e710-498b-90e9-7a846f16504b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 14:57:31.801616 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:31.801381    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0e02c795-e710-498b-90e9-7a846f16504b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0e02c795-e710-498b-90e9-7a846f16504b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 14:57:31.801616 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:31.801412    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0e02c795-e710-498b-90e9-7a846f16504b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0e02c795-e710-498b-90e9-7a846f16504b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 14:57:31.801616 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:31.801443    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0e02c795-e710-498b-90e9-7a846f16504b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0e02c795-e710-498b-90e9-7a846f16504b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 14:57:31.801616 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:31.801475    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0e02c795-e710-498b-90e9-7a846f16504b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0e02c795-e710-498b-90e9-7a846f16504b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 14:57:31.801616 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:31.801503    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0e02c795-e710-498b-90e9-7a846f16504b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0e02c795-e710-498b-90e9-7a846f16504b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 14:57:31.801616 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:31.801536    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0e02c795-e710-498b-90e9-7a846f16504b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0e02c795-e710-498b-90e9-7a846f16504b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 14:57:31.801616 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:31.801568    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0e02c795-e710-498b-90e9-7a846f16504b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0e02c795-e710-498b-90e9-7a846f16504b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 14:57:31.802069 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:31.801637    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0e02c795-e710-498b-90e9-7a846f16504b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0e02c795-e710-498b-90e9-7a846f16504b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 14:57:31.802069 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:31.801667    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0e02c795-e710-498b-90e9-7a846f16504b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0e02c795-e710-498b-90e9-7a846f16504b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 14:57:31.802069 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:31.801702    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0e02c795-e710-498b-90e9-7a846f16504b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0e02c795-e710-498b-90e9-7a846f16504b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 14:57:31.903988 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:31.902597    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0e02c795-e710-498b-90e9-7a846f16504b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0e02c795-e710-498b-90e9-7a846f16504b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 14:57:31.903988 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:31.902648    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0e02c795-e710-498b-90e9-7a846f16504b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0e02c795-e710-498b-90e9-7a846f16504b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 14:57:31.903988 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:31.902681    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0e02c795-e710-498b-90e9-7a846f16504b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0e02c795-e710-498b-90e9-7a846f16504b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 14:57:31.903988 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:31.902714    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0e02c795-e710-498b-90e9-7a846f16504b-web-config\") pod \"prometheus-k8s-0\" (UID: \"0e02c795-e710-498b-90e9-7a846f16504b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 14:57:31.903988 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:31.902745    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0e02c795-e710-498b-90e9-7a846f16504b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0e02c795-e710-498b-90e9-7a846f16504b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 14:57:31.903988 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:31.902772    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e02c795-e710-498b-90e9-7a846f16504b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0e02c795-e710-498b-90e9-7a846f16504b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 14:57:31.903988 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:31.902795    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0e02c795-e710-498b-90e9-7a846f16504b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0e02c795-e710-498b-90e9-7a846f16504b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 14:57:31.903988 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:31.902820    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e02c795-e710-498b-90e9-7a846f16504b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0e02c795-e710-498b-90e9-7a846f16504b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 14:57:31.903988 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:31.902844    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0e02c795-e710-498b-90e9-7a846f16504b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0e02c795-e710-498b-90e9-7a846f16504b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 14:57:31.903988 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:31.903170    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/0e02c795-e710-498b-90e9-7a846f16504b-config\") pod \"prometheus-k8s-0\" (UID: \"0e02c795-e710-498b-90e9-7a846f16504b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 14:57:31.903988 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:31.903257    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0e02c795-e710-498b-90e9-7a846f16504b-config-out\") pod \"prometheus-k8s-0\" (UID: \"0e02c795-e710-498b-90e9-7a846f16504b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 14:57:31.903988 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:31.903284    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e02c795-e710-498b-90e9-7a846f16504b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0e02c795-e710-498b-90e9-7a846f16504b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 14:57:31.903988 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:31.903310    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f68f6\" (UniqueName: \"kubernetes.io/projected/0e02c795-e710-498b-90e9-7a846f16504b-kube-api-access-f68f6\") pod \"prometheus-k8s-0\" (UID: \"0e02c795-e710-498b-90e9-7a846f16504b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 14:57:31.903988 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:31.903348    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0e02c795-e710-498b-90e9-7a846f16504b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0e02c795-e710-498b-90e9-7a846f16504b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 14:57:31.903988 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:31.903378    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0e02c795-e710-498b-90e9-7a846f16504b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0e02c795-e710-498b-90e9-7a846f16504b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 14:57:31.903988 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:31.903411    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0e02c795-e710-498b-90e9-7a846f16504b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0e02c795-e710-498b-90e9-7a846f16504b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 14:57:31.903988 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:31.903444    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0e02c795-e710-498b-90e9-7a846f16504b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0e02c795-e710-498b-90e9-7a846f16504b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 14:57:31.905017 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:31.903472    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0e02c795-e710-498b-90e9-7a846f16504b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0e02c795-e710-498b-90e9-7a846f16504b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 14:57:31.905017 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:31.903505    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0e02c795-e710-498b-90e9-7a846f16504b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0e02c795-e710-498b-90e9-7a846f16504b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 14:57:31.905017 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:31.903539    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0e02c795-e710-498b-90e9-7a846f16504b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0e02c795-e710-498b-90e9-7a846f16504b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 14:57:31.905017 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:31.903665    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0e02c795-e710-498b-90e9-7a846f16504b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0e02c795-e710-498b-90e9-7a846f16504b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 14:57:31.907190 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:31.907154    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0e02c795-e710-498b-90e9-7a846f16504b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0e02c795-e710-498b-90e9-7a846f16504b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 14:57:31.908489 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:31.908460    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e02c795-e710-498b-90e9-7a846f16504b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0e02c795-e710-498b-90e9-7a846f16504b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 14:57:31.909558 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:31.909178    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0e02c795-e710-498b-90e9-7a846f16504b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0e02c795-e710-498b-90e9-7a846f16504b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 14:57:31.910936 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:31.910912    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0e02c795-e710-498b-90e9-7a846f16504b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0e02c795-e710-498b-90e9-7a846f16504b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 14:57:31.912253 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:31.912205    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e02c795-e710-498b-90e9-7a846f16504b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0e02c795-e710-498b-90e9-7a846f16504b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 14:57:31.915154 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:31.913106    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0e02c795-e710-498b-90e9-7a846f16504b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0e02c795-e710-498b-90e9-7a846f16504b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 14:57:31.915154 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:31.913635    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0e02c795-e710-498b-90e9-7a846f16504b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0e02c795-e710-498b-90e9-7a846f16504b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 14:57:31.915154 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:31.914204    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e02c795-e710-498b-90e9-7a846f16504b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0e02c795-e710-498b-90e9-7a846f16504b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 14:57:31.917810 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:31.917743    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/0e02c795-e710-498b-90e9-7a846f16504b-config\") pod \"prometheus-k8s-0\" (UID: \"0e02c795-e710-498b-90e9-7a846f16504b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 14:57:31.922973 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:31.922920    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0e02c795-e710-498b-90e9-7a846f16504b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0e02c795-e710-498b-90e9-7a846f16504b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 14:57:31.925540 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:31.925503    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f68f6\" (UniqueName: \"kubernetes.io/projected/0e02c795-e710-498b-90e9-7a846f16504b-kube-api-access-f68f6\") pod \"prometheus-k8s-0\" (UID: \"0e02c795-e710-498b-90e9-7a846f16504b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 14:57:31.926317 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:31.926268    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0e02c795-e710-498b-90e9-7a846f16504b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0e02c795-e710-498b-90e9-7a846f16504b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 14:57:31.926420 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:31.926323    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0e02c795-e710-498b-90e9-7a846f16504b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0e02c795-e710-498b-90e9-7a846f16504b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 14:57:31.926989 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:31.926772    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0e02c795-e710-498b-90e9-7a846f16504b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0e02c795-e710-498b-90e9-7a846f16504b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 14:57:31.926989 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:31.926904    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0e02c795-e710-498b-90e9-7a846f16504b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0e02c795-e710-498b-90e9-7a846f16504b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 14:57:31.927151 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:31.927131    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0e02c795-e710-498b-90e9-7a846f16504b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0e02c795-e710-498b-90e9-7a846f16504b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 14:57:31.927696 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:31.927630    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0e02c795-e710-498b-90e9-7a846f16504b-web-config\") pod \"prometheus-k8s-0\" (UID: \"0e02c795-e710-498b-90e9-7a846f16504b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 14:57:31.930262 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:31.930154    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0e02c795-e710-498b-90e9-7a846f16504b-config-out\") pod \"prometheus-k8s-0\" (UID: \"0e02c795-e710-498b-90e9-7a846f16504b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 14:57:31.934148 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:31.934102    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0e02c795-e710-498b-90e9-7a846f16504b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0e02c795-e710-498b-90e9-7a846f16504b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 14:57:32.016639 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:32.016449    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3e1cce8d-f0cf-4f0e-ae15-1365159a053a","Type":"ContainerStarted","Data":"bc7906f2c8012f2038f22e05049d9de909599e278251977230501c7f660bd16e"}
Jul 23 14:57:32.025298 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:32.024845    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-5bc67dcd46-rzmqx" event={"ID":"ea3c4e5e-ccb4-417f-a9d9-5edc359ea988","Type":"ContainerStarted","Data":"6c1d50f6429d0255f0e294ee3da51f5ba503848b7f311bc2ac5f2030ca90d9c7"}
Jul 23 14:57:32.025689 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:32.025554    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-5bc67dcd46-rzmqx"
Jul 23 14:57:32.028410 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:32.028376    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-29hwq" event={"ID":"e1ee8253-0c66-4e49-827b-42d03d36c796","Type":"ContainerStarted","Data":"abe1deb6195b3399e410d1d1bb5e059e53e81a4148858320479cc20cf9089ad5"}
Jul 23 14:57:32.035511 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:32.035283    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 14:57:32.043609 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:32.043529    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-5bc67dcd46-rzmqx"
Jul 23 14:57:32.046426 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:32.045990    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-5bc67dcd46-rzmqx" podStartSLOduration=1.88230106 podStartE2EDuration="19.045967995s" totalImagesPullingTime="17.163666935s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 14:57:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 14:57:32.044031782 +0000 UTC m=+293.551905531" watchObservedRunningTime="2026-07-23 14:57:32.045967995 +0000 UTC m=+293.553841744"
Jul 23 14:57:32.059225 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:57:32.059174    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-hvw8v" podUID="413d2464-7258-4246-85bb-ac9f03976ee5"
Jul 23 14:57:32.282937 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:32.282797    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 23 14:57:32.750371 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:57:32.750327    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e02c795_e710_498b_90e9_7a846f16504b.slice/crio-2cc6f70b1c351703c3c59d998c231fbeb4311d82cf61cd78593a6e5587d5e662 WatchSource:0}: Error finding container 2cc6f70b1c351703c3c59d998c231fbeb4311d82cf61cd78593a6e5587d5e662: Status 404 returned error can't find the container with id 2cc6f70b1c351703c3c59d998c231fbeb4311d82cf61cd78593a6e5587d5e662
Jul 23 14:57:33.043658 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:33.043537    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0e02c795-e710-498b-90e9-7a846f16504b","Type":"ContainerStarted","Data":"2cc6f70b1c351703c3c59d998c231fbeb4311d82cf61cd78593a6e5587d5e662"}
Jul 23 14:57:33.700651 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:33.700614    2560 patch_prober.go:28] interesting pod/image-registry-5cf48ff8c9-p9xlr container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 23 14:57:33.700863 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:33.700678    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-5cf48ff8c9-p9xlr" podUID="7df58b86-f921-4316-b8b1-f48137124a76" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 23 14:57:36.058706 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:36.058672    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-scspg" event={"ID":"a2a2b51d-3d22-42dd-96b6-093bdf7c3f7a","Type":"ContainerStarted","Data":"02eb203436b5793e4d6f79b4dceffb631a978056f88007094f70fc2eb68f72ef"}
Jul 23 14:57:36.059168 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:36.058735    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-scspg" event={"ID":"a2a2b51d-3d22-42dd-96b6-093bdf7c3f7a","Type":"ContainerStarted","Data":"46c5c1fd898aefca9cecedd8e89c30d2baba5467cc05542144fa93fc42ef245c"}
Jul 23 14:57:36.059168 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:36.058790    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-scspg"
Jul 23 14:57:36.060808 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:36.060781    2560 generic.go:356] "Generic (PLEG): container finished" podID="a2e2585d-0204-4faa-807e-b1e28263c9bf" containerID="28ae000b0f20c1e1b023a5df215d371c02cf9aa41e18eccee59327cb61da02cd" exitCode=0
Jul 23 14:57:36.060934 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:36.060879    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wt7xl" event={"ID":"a2e2585d-0204-4faa-807e-b1e28263c9bf","Type":"ContainerDied","Data":"28ae000b0f20c1e1b023a5df215d371c02cf9aa41e18eccee59327cb61da02cd"}
Jul 23 14:57:36.062867 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:36.062842    2560 generic.go:356] "Generic (PLEG): container finished" podID="3e1cce8d-f0cf-4f0e-ae15-1365159a053a" containerID="9a8f3465b23cd42e0d6ab40d06c9d912ab160210b2ebc175fd262cae044995ad" exitCode=0
Jul 23 14:57:36.062983 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:36.062935    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3e1cce8d-f0cf-4f0e-ae15-1365159a053a","Type":"ContainerDied","Data":"9a8f3465b23cd42e0d6ab40d06c9d912ab160210b2ebc175fd262cae044995ad"}
Jul 23 14:57:36.064879 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:36.064834    2560 generic.go:356] "Generic (PLEG): container finished" podID="0e02c795-e710-498b-90e9-7a846f16504b" containerID="81d8a68383821ddfb67b911a924c49baadd716f0cc43e42e004673cae527de01" exitCode=0
Jul 23 14:57:36.064981 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:36.064920    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0e02c795-e710-498b-90e9-7a846f16504b","Type":"ContainerDied","Data":"81d8a68383821ddfb67b911a924c49baadd716f0cc43e42e004673cae527de01"}
Jul 23 14:57:36.066711 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:36.066629    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-29hwq" event={"ID":"e1ee8253-0c66-4e49-827b-42d03d36c796","Type":"ContainerStarted","Data":"ab412cae31c991fbbc2c1f570d3a273e778799a2de9aeb25f0f013ede21b2cee"}
Jul 23 14:57:36.067063 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:36.067040    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-29hwq"
Jul 23 14:57:36.073446 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:36.073421    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-29hwq"
Jul 23 14:57:36.077560 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:36.077518    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-scspg" podStartSLOduration=251.737631931 podStartE2EDuration="4m23.077499698s" totalImagesPullingTime="11.339867767s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 14:53:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 14:57:36.076035514 +0000 UTC m=+297.583909262" watchObservedRunningTime="2026-07-23 14:57:36.077499698 +0000 UTC m=+297.585373445"
Jul 23 14:57:36.169106 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:36.169053    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-29hwq" podStartSLOduration=2.36072655 podStartE2EDuration="6.169036015s" totalImagesPullingTime="3.808309465s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 14:57:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 14:57:36.167822311 +0000 UTC m=+297.675696060" watchObservedRunningTime="2026-07-23 14:57:36.169036015 +0000 UTC m=+297.676909781"
Jul 23 14:57:37.079446 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:37.079361    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wt7xl" event={"ID":"a2e2585d-0204-4faa-807e-b1e28263c9bf","Type":"ContainerStarted","Data":"5d45e7343f89ffeacb983c171523985829072ba38ed5ce8af99000f98e8900ff"}
Jul 23 14:57:37.080067 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:37.079621    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wt7xl" event={"ID":"a2e2585d-0204-4faa-807e-b1e28263c9bf","Type":"ContainerStarted","Data":"41f28d61069584ea42c3cd7c3562099c29cdea7651a139af5fe828aa6e521659"}
Jul 23 14:57:37.103501 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:37.103427    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-wt7xl" podStartSLOduration=7.890097196 podStartE2EDuration="12.103408101s" totalImagesPullingTime="4.074205555s" totalInitContainerRuntime="139.10535ms" isStatefulPod=true podCreationTimestamp="2026-07-23 14:57:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 14:57:37.101769539 +0000 UTC m=+298.609643288" watchObservedRunningTime="2026-07-23 14:57:37.103408101 +0000 UTC m=+298.611281850"
Jul 23 14:57:38.719507 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:38.719460    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5cf48ff8c9-p9xlr" podUID="7df58b86-f921-4316-b8b1-f48137124a76" containerName="registry" containerID="cri-o://9d1a6350541d81d6effeb58cd65c5da76043d8af9823280770ac8cdb92966cd3" gracePeriod=30
Jul 23 14:57:40.096322 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:40.096285    2560 generic.go:356] "Generic (PLEG): container finished" podID="7df58b86-f921-4316-b8b1-f48137124a76" containerID="9d1a6350541d81d6effeb58cd65c5da76043d8af9823280770ac8cdb92966cd3" exitCode=0
Jul 23 14:57:40.096799 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:40.096365    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5cf48ff8c9-p9xlr" event={"ID":"7df58b86-f921-4316-b8b1-f48137124a76","Type":"ContainerDied","Data":"9d1a6350541d81d6effeb58cd65c5da76043d8af9823280770ac8cdb92966cd3"}
Jul 23 14:57:40.102631 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:40.102598    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z4fvn_e0c66f26-06fe-4273-ba29-fa20ce89b873/ovn-acl-logging/0.log"
Jul 23 14:57:40.103218 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:40.103199    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z4fvn_e0c66f26-06fe-4273-ba29-fa20ce89b873/ovn-acl-logging/0.log"
Jul 23 14:57:40.113751 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:40.113716    2560 kubelet.go:1648] "Image garbage collection succeeded"
Jul 23 14:57:40.279494 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:40.279461    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5cf48ff8c9-p9xlr"
Jul 23 14:57:40.389639 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:40.389611    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7df58b86-f921-4316-b8b1-f48137124a76-ca-trust-extracted\") pod \"7df58b86-f921-4316-b8b1-f48137124a76\" (UID: \"7df58b86-f921-4316-b8b1-f48137124a76\") "
Jul 23 14:57:40.389801 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:40.389664    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9fjj5\" (UniqueName: \"kubernetes.io/projected/7df58b86-f921-4316-b8b1-f48137124a76-kube-api-access-9fjj5\") pod \"7df58b86-f921-4316-b8b1-f48137124a76\" (UID: \"7df58b86-f921-4316-b8b1-f48137124a76\") "
Jul 23 14:57:40.389801 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:40.389702    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7df58b86-f921-4316-b8b1-f48137124a76-installation-pull-secrets\") pod \"7df58b86-f921-4316-b8b1-f48137124a76\" (UID: \"7df58b86-f921-4316-b8b1-f48137124a76\") "
Jul 23 14:57:40.389801 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:40.389740    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7df58b86-f921-4316-b8b1-f48137124a76-registry-tls\") pod \"7df58b86-f921-4316-b8b1-f48137124a76\" (UID: \"7df58b86-f921-4316-b8b1-f48137124a76\") "
Jul 23 14:57:40.389801 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:40.389796    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7df58b86-f921-4316-b8b1-f48137124a76-image-registry-private-configuration\") pod \"7df58b86-f921-4316-b8b1-f48137124a76\" (UID: \"7df58b86-f921-4316-b8b1-f48137124a76\") "
Jul 23 14:57:40.390008 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:40.389831    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7df58b86-f921-4316-b8b1-f48137124a76-bound-sa-token\") pod \"7df58b86-f921-4316-b8b1-f48137124a76\" (UID: \"7df58b86-f921-4316-b8b1-f48137124a76\") "
Jul 23 14:57:40.390008 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:40.389863    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7df58b86-f921-4316-b8b1-f48137124a76-trusted-ca\") pod \"7df58b86-f921-4316-b8b1-f48137124a76\" (UID: \"7df58b86-f921-4316-b8b1-f48137124a76\") "
Jul 23 14:57:40.390008 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:40.389912    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7df58b86-f921-4316-b8b1-f48137124a76-registry-certificates\") pod \"7df58b86-f921-4316-b8b1-f48137124a76\" (UID: \"7df58b86-f921-4316-b8b1-f48137124a76\") "
Jul 23 14:57:40.390638 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:40.390502    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7df58b86-f921-4316-b8b1-f48137124a76-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "7df58b86-f921-4316-b8b1-f48137124a76" (UID: "7df58b86-f921-4316-b8b1-f48137124a76"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 14:57:40.390638 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:40.390603    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7df58b86-f921-4316-b8b1-f48137124a76-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "7df58b86-f921-4316-b8b1-f48137124a76" (UID: "7df58b86-f921-4316-b8b1-f48137124a76"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 14:57:40.393117 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:40.393084    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7df58b86-f921-4316-b8b1-f48137124a76-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "7df58b86-f921-4316-b8b1-f48137124a76" (UID: "7df58b86-f921-4316-b8b1-f48137124a76"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 14:57:40.393117 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:40.393100    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7df58b86-f921-4316-b8b1-f48137124a76-kube-api-access-9fjj5" (OuterVolumeSpecName: "kube-api-access-9fjj5") pod "7df58b86-f921-4316-b8b1-f48137124a76" (UID: "7df58b86-f921-4316-b8b1-f48137124a76"). InnerVolumeSpecName "kube-api-access-9fjj5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 14:57:40.393117 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:40.393100    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7df58b86-f921-4316-b8b1-f48137124a76-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "7df58b86-f921-4316-b8b1-f48137124a76" (UID: "7df58b86-f921-4316-b8b1-f48137124a76"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 14:57:40.393351 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:40.393153    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7df58b86-f921-4316-b8b1-f48137124a76-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "7df58b86-f921-4316-b8b1-f48137124a76" (UID: "7df58b86-f921-4316-b8b1-f48137124a76"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 14:57:40.395028 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:40.394976    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7df58b86-f921-4316-b8b1-f48137124a76-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "7df58b86-f921-4316-b8b1-f48137124a76" (UID: "7df58b86-f921-4316-b8b1-f48137124a76"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 14:57:40.401276 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:40.401224    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7df58b86-f921-4316-b8b1-f48137124a76-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "7df58b86-f921-4316-b8b1-f48137124a76" (UID: "7df58b86-f921-4316-b8b1-f48137124a76"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 14:57:40.490757 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:40.490723    2560 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7df58b86-f921-4316-b8b1-f48137124a76-installation-pull-secrets\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Jul 23 14:57:40.490757 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:40.490750    2560 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7df58b86-f921-4316-b8b1-f48137124a76-registry-tls\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Jul 23 14:57:40.490757 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:40.490760    2560 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7df58b86-f921-4316-b8b1-f48137124a76-image-registry-private-configuration\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Jul 23 14:57:40.490975 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:40.490772    2560 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7df58b86-f921-4316-b8b1-f48137124a76-bound-sa-token\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Jul 23 14:57:40.490975 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:40.490785    2560 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7df58b86-f921-4316-b8b1-f48137124a76-trusted-ca\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Jul 23 14:57:40.490975 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:40.490797    2560 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7df58b86-f921-4316-b8b1-f48137124a76-registry-certificates\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Jul 23 14:57:40.490975 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:40.490810    2560 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7df58b86-f921-4316-b8b1-f48137124a76-ca-trust-extracted\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Jul 23 14:57:40.490975 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:40.490822    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9fjj5\" (UniqueName: \"kubernetes.io/projected/7df58b86-f921-4316-b8b1-f48137124a76-kube-api-access-9fjj5\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Jul 23 14:57:40.813019 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:40.812295    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 23 14:57:41.103979 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:41.103897    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3e1cce8d-f0cf-4f0e-ae15-1365159a053a","Type":"ContainerStarted","Data":"b64f86bef3a49309851ed097ac04545ab918535cd3ff4ed0ab2386becedf3cd4"}
Jul 23 14:57:41.104378 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:41.103982    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3e1cce8d-f0cf-4f0e-ae15-1365159a053a","Type":"ContainerStarted","Data":"db0501a851762d35ac6e4dde6c9402245ef142857a0d3eb876a513817e2a00ad"}
Jul 23 14:57:41.104378 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:41.104026    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3e1cce8d-f0cf-4f0e-ae15-1365159a053a","Type":"ContainerStarted","Data":"397bfb664ee74066daa96e163f418d3ea31901cfb3663cbb312dfffde499d025"}
Jul 23 14:57:41.104378 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:41.104063    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3e1cce8d-f0cf-4f0e-ae15-1365159a053a","Type":"ContainerStarted","Data":"2247b9bdb6272f07bba16a3fc55671f8d9812b11fc065b0e286fc581eb338127"}
Jul 23 14:57:41.104378 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:41.104100    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3e1cce8d-f0cf-4f0e-ae15-1365159a053a","Type":"ContainerStarted","Data":"c3856691cff2e572946b1dec6433343ed6ca9e88cb2b15d25568d7fbe38f0aa2"}
Jul 23 14:57:41.106152 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:41.106125    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0e02c795-e710-498b-90e9-7a846f16504b","Type":"ContainerStarted","Data":"68b9f976fae08385ab6b37ea178f3abecde29cd40a180a0ee35e55caf2442a69"}
Jul 23 14:57:41.106307 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:41.106199    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0e02c795-e710-498b-90e9-7a846f16504b","Type":"ContainerStarted","Data":"fb9a82dce1d1218395b7d51b4ceaf2dd7a35bcd1335bb1785db44de2a013baca"}
Jul 23 14:57:41.107465 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:41.107444    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5cf48ff8c9-p9xlr" event={"ID":"7df58b86-f921-4316-b8b1-f48137124a76","Type":"ContainerDied","Data":"b7326f51cc138e2aafaeb77b583933a9876fff66ec7f1454c1fa9899f3d74bc1"}
Jul 23 14:57:41.107579 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:41.107486    2560 scope.go:117] "RemoveContainer" containerID="9d1a6350541d81d6effeb58cd65c5da76043d8af9823280770ac8cdb92966cd3"
Jul 23 14:57:41.107579 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:41.107448    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5cf48ff8c9-p9xlr"
Jul 23 14:57:41.125165 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:41.125134    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5cf48ff8c9-p9xlr"]
Jul 23 14:57:41.128593 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:41.128565    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5cf48ff8c9-p9xlr"]
Jul 23 14:57:43.062011 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:43.061979    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7df58b86-f921-4316-b8b1-f48137124a76" path="/var/lib/kubelet/pods/7df58b86-f921-4316-b8b1-f48137124a76/volumes"
Jul 23 14:57:43.125365 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:43.125331    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3e1cce8d-f0cf-4f0e-ae15-1365159a053a","Type":"ContainerStarted","Data":"816d67dfe01c9d1463bd704f5f3e7eb76d8a3c12dabdfb0f0a235826aff7da21"}
Jul 23 14:57:43.153692 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:43.153638    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=8.141149525 podStartE2EDuration="17.153622083s" totalImagesPullingTime="8.96535266s" totalInitContainerRuntime="47.119898ms" isStatefulPod=false podCreationTimestamp="2026-07-23 14:57:26 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 14:57:43.153063106 +0000 UTC m=+304.660936855" watchObservedRunningTime="2026-07-23 14:57:43.153622083 +0000 UTC m=+304.661495831"
Jul 23 14:57:44.131907 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:44.131868    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0e02c795-e710-498b-90e9-7a846f16504b","Type":"ContainerStarted","Data":"e85bdce164ebac8a98ec711f7add11f2fad6a75a9e923c57a536b8e9a8d82269"}
Jul 23 14:57:44.132414 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:44.131951    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0e02c795-e710-498b-90e9-7a846f16504b","Type":"ContainerStarted","Data":"582a86367ba1729f8aa11b25481bab1dec45f569ba6dab4828cde9b5a69c48bb"}
Jul 23 14:57:44.132414 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:44.132001    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0e02c795-e710-498b-90e9-7a846f16504b","Type":"ContainerStarted","Data":"5e2ae07470542d1ef4168a8d5182463c0c9bb4efa031ca33463962a9a5dbe9f8"}
Jul 23 14:57:44.132414 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:44.132026    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0e02c795-e710-498b-90e9-7a846f16504b","Type":"ContainerStarted","Data":"c1b2025ca2641246eb41918b12b99a699820606d0486afdc0387fc52768c0be6"}
Jul 23 14:57:44.162055 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:44.161972    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.60087093 podStartE2EDuration="13.161951321s" totalImagesPullingTime="9.543792849s" totalInitContainerRuntime="17.287542ms" isStatefulPod=false podCreationTimestamp="2026-07-23 14:57:31 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 14:57:44.159771408 +0000 UTC m=+305.667645157" watchObservedRunningTime="2026-07-23 14:57:44.161951321 +0000 UTC m=+305.669825070"
Jul 23 14:57:46.057198 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:46.057158    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hvw8v"
Jul 23 14:57:46.082379 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:46.082349    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-scspg"
Jul 23 14:57:47.035642 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:47.035593    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 14:57:56.005381 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:56.005327    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-cbdfb4fbf-nl6bn" podUID="a37889b3-f376-4734-b5d4-812cefd9c0fc" containerName="console" containerID="cri-o://109029d5789480bb59189c57a1512683251e8dfe61c943141d8aab45c895713a" gracePeriod=15
Jul 23 14:57:56.180477 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:56.180437    2560 generic.go:356] "Generic (PLEG): container finished" podID="a37889b3-f376-4734-b5d4-812cefd9c0fc" containerID="109029d5789480bb59189c57a1512683251e8dfe61c943141d8aab45c895713a" exitCode=0
Jul 23 14:57:56.180638 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:56.180508    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cbdfb4fbf-nl6bn" event={"ID":"a37889b3-f376-4734-b5d4-812cefd9c0fc","Type":"ContainerDied","Data":"109029d5789480bb59189c57a1512683251e8dfe61c943141d8aab45c895713a"}
Jul 23 14:57:56.289916 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:56.289886    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cbdfb4fbf-nl6bn"
Jul 23 14:57:56.328069 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:56.328038    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a37889b3-f376-4734-b5d4-812cefd9c0fc-console-oauth-config\") pod \"a37889b3-f376-4734-b5d4-812cefd9c0fc\" (UID: \"a37889b3-f376-4734-b5d4-812cefd9c0fc\") "
Jul 23 14:57:56.328261 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:56.328098    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a37889b3-f376-4734-b5d4-812cefd9c0fc-oauth-serving-cert\") pod \"a37889b3-f376-4734-b5d4-812cefd9c0fc\" (UID: \"a37889b3-f376-4734-b5d4-812cefd9c0fc\") "
Jul 23 14:57:56.328261 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:56.328132    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a37889b3-f376-4734-b5d4-812cefd9c0fc-service-ca\") pod \"a37889b3-f376-4734-b5d4-812cefd9c0fc\" (UID: \"a37889b3-f376-4734-b5d4-812cefd9c0fc\") "
Jul 23 14:57:56.328261 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:56.328186    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6t2xz\" (UniqueName: \"kubernetes.io/projected/a37889b3-f376-4734-b5d4-812cefd9c0fc-kube-api-access-6t2xz\") pod \"a37889b3-f376-4734-b5d4-812cefd9c0fc\" (UID: \"a37889b3-f376-4734-b5d4-812cefd9c0fc\") "
Jul 23 14:57:56.328261 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:56.328209    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a37889b3-f376-4734-b5d4-812cefd9c0fc-console-config\") pod \"a37889b3-f376-4734-b5d4-812cefd9c0fc\" (UID: \"a37889b3-f376-4734-b5d4-812cefd9c0fc\") "
Jul 23 14:57:56.328261 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:56.328260    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a37889b3-f376-4734-b5d4-812cefd9c0fc-console-serving-cert\") pod \"a37889b3-f376-4734-b5d4-812cefd9c0fc\" (UID: \"a37889b3-f376-4734-b5d4-812cefd9c0fc\") "
Jul 23 14:57:56.328550 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:56.328519    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a37889b3-f376-4734-b5d4-812cefd9c0fc-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "a37889b3-f376-4734-b5d4-812cefd9c0fc" (UID: "a37889b3-f376-4734-b5d4-812cefd9c0fc"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 14:57:56.328615 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:56.328576    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a37889b3-f376-4734-b5d4-812cefd9c0fc-service-ca" (OuterVolumeSpecName: "service-ca") pod "a37889b3-f376-4734-b5d4-812cefd9c0fc" (UID: "a37889b3-f376-4734-b5d4-812cefd9c0fc"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 14:57:56.328870 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:56.328845    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a37889b3-f376-4734-b5d4-812cefd9c0fc-console-config" (OuterVolumeSpecName: "console-config") pod "a37889b3-f376-4734-b5d4-812cefd9c0fc" (UID: "a37889b3-f376-4734-b5d4-812cefd9c0fc"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 14:57:56.330453 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:56.330431    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a37889b3-f376-4734-b5d4-812cefd9c0fc-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "a37889b3-f376-4734-b5d4-812cefd9c0fc" (UID: "a37889b3-f376-4734-b5d4-812cefd9c0fc"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 14:57:56.330535 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:56.330504    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a37889b3-f376-4734-b5d4-812cefd9c0fc-kube-api-access-6t2xz" (OuterVolumeSpecName: "kube-api-access-6t2xz") pod "a37889b3-f376-4734-b5d4-812cefd9c0fc" (UID: "a37889b3-f376-4734-b5d4-812cefd9c0fc"). InnerVolumeSpecName "kube-api-access-6t2xz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 14:57:56.330668 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:56.330640    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a37889b3-f376-4734-b5d4-812cefd9c0fc-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "a37889b3-f376-4734-b5d4-812cefd9c0fc" (UID: "a37889b3-f376-4734-b5d4-812cefd9c0fc"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 14:57:56.429123 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:56.429082    2560 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a37889b3-f376-4734-b5d4-812cefd9c0fc-service-ca\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Jul 23 14:57:56.429123 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:56.429116    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6t2xz\" (UniqueName: \"kubernetes.io/projected/a37889b3-f376-4734-b5d4-812cefd9c0fc-kube-api-access-6t2xz\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Jul 23 14:57:56.429123 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:56.429130    2560 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a37889b3-f376-4734-b5d4-812cefd9c0fc-console-config\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Jul 23 14:57:56.429379 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:56.429142    2560 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a37889b3-f376-4734-b5d4-812cefd9c0fc-console-serving-cert\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Jul 23 14:57:56.429379 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:56.429154    2560 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a37889b3-f376-4734-b5d4-812cefd9c0fc-console-oauth-config\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Jul 23 14:57:56.429379 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:56.429165    2560 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a37889b3-f376-4734-b5d4-812cefd9c0fc-oauth-serving-cert\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Jul 23 14:57:57.186021 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:57.185990    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cbdfb4fbf-nl6bn"
Jul 23 14:57:57.186513 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:57.185994    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cbdfb4fbf-nl6bn" event={"ID":"a37889b3-f376-4734-b5d4-812cefd9c0fc","Type":"ContainerDied","Data":"52b3b1943e9c67d37f46f4929ef18aeb3a622ab770f378a6190921fad1e558e7"}
Jul 23 14:57:57.186513 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:57.186172    2560 scope.go:117] "RemoveContainer" containerID="109029d5789480bb59189c57a1512683251e8dfe61c943141d8aab45c895713a"
Jul 23 14:57:57.206956 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:57.206925    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-cbdfb4fbf-nl6bn"]
Jul 23 14:57:57.213810 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:57.213782    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-cbdfb4fbf-nl6bn"]
Jul 23 14:57:59.061480 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:57:59.061446    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a37889b3-f376-4734-b5d4-812cefd9c0fc" path="/var/lib/kubelet/pods/a37889b3-f376-4734-b5d4-812cefd9c0fc/volumes"
Jul 23 14:58:19.267965 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:19.267933    2560 generic.go:356] "Generic (PLEG): container finished" podID="9ae597eb-68e0-447c-bf0c-781a80684af1" containerID="aa8036d24136eb896ca9348b78a1672d041699f8716fd9f3eb00fdeaf51688d1" exitCode=0
Jul 23 14:58:19.268409 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:19.267996    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-k5zd5" event={"ID":"9ae597eb-68e0-447c-bf0c-781a80684af1","Type":"ContainerDied","Data":"aa8036d24136eb896ca9348b78a1672d041699f8716fd9f3eb00fdeaf51688d1"}
Jul 23 14:58:19.268409 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:19.268332    2560 scope.go:117] "RemoveContainer" containerID="aa8036d24136eb896ca9348b78a1672d041699f8716fd9f3eb00fdeaf51688d1"
Jul 23 14:58:20.273530 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:20.273500    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-k5zd5" event={"ID":"9ae597eb-68e0-447c-bf0c-781a80684af1","Type":"ContainerStarted","Data":"6f7f6614f27d9d9c02a780057580e8d76f7f34134e860b9b5ef98c23858a3bd6"}
Jul 23 14:58:23.286142 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:23.286105    2560 generic.go:356] "Generic (PLEG): container finished" podID="a253789c-c1c0-409f-b335-63fe5f2fb4af" containerID="04382b3482c980cae213d693d100204db33b50bd25cf42774bc181b7ec648ed3" exitCode=0
Jul 23 14:58:23.286556 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:23.286177    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-zndrp" event={"ID":"a253789c-c1c0-409f-b335-63fe5f2fb4af","Type":"ContainerDied","Data":"04382b3482c980cae213d693d100204db33b50bd25cf42774bc181b7ec648ed3"}
Jul 23 14:58:23.286556 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:23.286552    2560 scope.go:117] "RemoveContainer" containerID="04382b3482c980cae213d693d100204db33b50bd25cf42774bc181b7ec648ed3"
Jul 23 14:58:24.294577 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:24.294544    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-zndrp" event={"ID":"a253789c-c1c0-409f-b335-63fe5f2fb4af","Type":"ContainerStarted","Data":"f530681441d6677d237ea9c291673577064c1468694f4ef46f33432ea3c12b30"}
Jul 23 14:58:32.036438 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:32.036342    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 14:58:32.056458 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:32.056430    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 14:58:32.341548 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:32.341469    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 14:58:45.803445 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:45.803412    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 23 14:58:45.803838 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:45.803698    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="3e1cce8d-f0cf-4f0e-ae15-1365159a053a" containerName="alertmanager" containerID="cri-o://c3856691cff2e572946b1dec6433343ed6ca9e88cb2b15d25568d7fbe38f0aa2" gracePeriod=120
Jul 23 14:58:45.803838 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:45.803760    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="3e1cce8d-f0cf-4f0e-ae15-1365159a053a" containerName="kube-rbac-proxy-metric" containerID="cri-o://b64f86bef3a49309851ed097ac04545ab918535cd3ff4ed0ab2386becedf3cd4" gracePeriod=120
Jul 23 14:58:45.803953 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:45.803808    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="3e1cce8d-f0cf-4f0e-ae15-1365159a053a" containerName="prom-label-proxy" containerID="cri-o://816d67dfe01c9d1463bd704f5f3e7eb76d8a3c12dabdfb0f0a235826aff7da21" gracePeriod=120
Jul 23 14:58:45.803953 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:45.803850    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="3e1cce8d-f0cf-4f0e-ae15-1365159a053a" containerName="config-reloader" containerID="cri-o://2247b9bdb6272f07bba16a3fc55671f8d9812b11fc065b0e286fc581eb338127" gracePeriod=120
Jul 23 14:58:45.803953 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:45.803890    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="3e1cce8d-f0cf-4f0e-ae15-1365159a053a" containerName="kube-rbac-proxy" containerID="cri-o://db0501a851762d35ac6e4dde6c9402245ef142857a0d3eb876a513817e2a00ad" gracePeriod=120
Jul 23 14:58:45.804097 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:45.803806    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="3e1cce8d-f0cf-4f0e-ae15-1365159a053a" containerName="kube-rbac-proxy-web" containerID="cri-o://397bfb664ee74066daa96e163f418d3ea31901cfb3663cbb312dfffde499d025" gracePeriod=120
Jul 23 14:58:46.379336 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:46.379301    2560 generic.go:356] "Generic (PLEG): container finished" podID="3e1cce8d-f0cf-4f0e-ae15-1365159a053a" containerID="816d67dfe01c9d1463bd704f5f3e7eb76d8a3c12dabdfb0f0a235826aff7da21" exitCode=0
Jul 23 14:58:46.379336 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:46.379327    2560 generic.go:356] "Generic (PLEG): container finished" podID="3e1cce8d-f0cf-4f0e-ae15-1365159a053a" containerID="db0501a851762d35ac6e4dde6c9402245ef142857a0d3eb876a513817e2a00ad" exitCode=0
Jul 23 14:58:46.379336 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:46.379333    2560 generic.go:356] "Generic (PLEG): container finished" podID="3e1cce8d-f0cf-4f0e-ae15-1365159a053a" containerID="2247b9bdb6272f07bba16a3fc55671f8d9812b11fc065b0e286fc581eb338127" exitCode=0
Jul 23 14:58:46.379336 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:46.379338    2560 generic.go:356] "Generic (PLEG): container finished" podID="3e1cce8d-f0cf-4f0e-ae15-1365159a053a" containerID="c3856691cff2e572946b1dec6433343ed6ca9e88cb2b15d25568d7fbe38f0aa2" exitCode=0
Jul 23 14:58:46.379589 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:46.379372    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3e1cce8d-f0cf-4f0e-ae15-1365159a053a","Type":"ContainerDied","Data":"816d67dfe01c9d1463bd704f5f3e7eb76d8a3c12dabdfb0f0a235826aff7da21"}
Jul 23 14:58:46.379589 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:46.379435    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3e1cce8d-f0cf-4f0e-ae15-1365159a053a","Type":"ContainerDied","Data":"db0501a851762d35ac6e4dde6c9402245ef142857a0d3eb876a513817e2a00ad"}
Jul 23 14:58:46.379589 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:46.379461    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3e1cce8d-f0cf-4f0e-ae15-1365159a053a","Type":"ContainerDied","Data":"2247b9bdb6272f07bba16a3fc55671f8d9812b11fc065b0e286fc581eb338127"}
Jul 23 14:58:46.379589 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:46.379485    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3e1cce8d-f0cf-4f0e-ae15-1365159a053a","Type":"ContainerDied","Data":"c3856691cff2e572946b1dec6433343ed6ca9e88cb2b15d25568d7fbe38f0aa2"}
Jul 23 14:58:47.044136 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.044102    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 23 14:58:47.071118 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.071095    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3e1cce8d-f0cf-4f0e-ae15-1365159a053a-web-config\") pod \"3e1cce8d-f0cf-4f0e-ae15-1365159a053a\" (UID: \"3e1cce8d-f0cf-4f0e-ae15-1365159a053a\") "
Jul 23 14:58:47.071310 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.071124    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3e1cce8d-f0cf-4f0e-ae15-1365159a053a-metrics-client-ca\") pod \"3e1cce8d-f0cf-4f0e-ae15-1365159a053a\" (UID: \"3e1cce8d-f0cf-4f0e-ae15-1365159a053a\") "
Jul 23 14:58:47.071310 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.071148    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/3e1cce8d-f0cf-4f0e-ae15-1365159a053a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"3e1cce8d-f0cf-4f0e-ae15-1365159a053a\" (UID: \"3e1cce8d-f0cf-4f0e-ae15-1365159a053a\") "
Jul 23 14:58:47.071310 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.071164    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3e1cce8d-f0cf-4f0e-ae15-1365159a053a-config-volume\") pod \"3e1cce8d-f0cf-4f0e-ae15-1365159a053a\" (UID: \"3e1cce8d-f0cf-4f0e-ae15-1365159a053a\") "
Jul 23 14:58:47.071310 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.071193    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3e1cce8d-f0cf-4f0e-ae15-1365159a053a-secret-alertmanager-kube-rbac-proxy-web\") pod \"3e1cce8d-f0cf-4f0e-ae15-1365159a053a\" (UID: \"3e1cce8d-f0cf-4f0e-ae15-1365159a053a\") "
Jul 23 14:58:47.071310 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.071213    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qbs9x\" (UniqueName: \"kubernetes.io/projected/3e1cce8d-f0cf-4f0e-ae15-1365159a053a-kube-api-access-qbs9x\") pod \"3e1cce8d-f0cf-4f0e-ae15-1365159a053a\" (UID: \"3e1cce8d-f0cf-4f0e-ae15-1365159a053a\") "
Jul 23 14:58:47.071310 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.071262    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3e1cce8d-f0cf-4f0e-ae15-1365159a053a-secret-alertmanager-main-tls\") pod \"3e1cce8d-f0cf-4f0e-ae15-1365159a053a\" (UID: \"3e1cce8d-f0cf-4f0e-ae15-1365159a053a\") "
Jul 23 14:58:47.071310 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.071280    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3e1cce8d-f0cf-4f0e-ae15-1365159a053a-secret-alertmanager-kube-rbac-proxy\") pod \"3e1cce8d-f0cf-4f0e-ae15-1365159a053a\" (UID: \"3e1cce8d-f0cf-4f0e-ae15-1365159a053a\") "
Jul 23 14:58:47.071310 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.071296    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/3e1cce8d-f0cf-4f0e-ae15-1365159a053a-alertmanager-main-db\") pod \"3e1cce8d-f0cf-4f0e-ae15-1365159a053a\" (UID: \"3e1cce8d-f0cf-4f0e-ae15-1365159a053a\") "
Jul 23 14:58:47.071743 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.071320    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3e1cce8d-f0cf-4f0e-ae15-1365159a053a-config-out\") pod \"3e1cce8d-f0cf-4f0e-ae15-1365159a053a\" (UID: \"3e1cce8d-f0cf-4f0e-ae15-1365159a053a\") "
Jul 23 14:58:47.071743 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.071375    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3e1cce8d-f0cf-4f0e-ae15-1365159a053a-alertmanager-trusted-ca-bundle\") pod \"3e1cce8d-f0cf-4f0e-ae15-1365159a053a\" (UID: \"3e1cce8d-f0cf-4f0e-ae15-1365159a053a\") "
Jul 23 14:58:47.071743 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.071405    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3e1cce8d-f0cf-4f0e-ae15-1365159a053a-tls-assets\") pod \"3e1cce8d-f0cf-4f0e-ae15-1365159a053a\" (UID: \"3e1cce8d-f0cf-4f0e-ae15-1365159a053a\") "
Jul 23 14:58:47.071743 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.071433    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3e1cce8d-f0cf-4f0e-ae15-1365159a053a-cluster-tls-config\") pod \"3e1cce8d-f0cf-4f0e-ae15-1365159a053a\" (UID: \"3e1cce8d-f0cf-4f0e-ae15-1365159a053a\") "
Jul 23 14:58:47.071743 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.071544    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e1cce8d-f0cf-4f0e-ae15-1365159a053a-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "3e1cce8d-f0cf-4f0e-ae15-1365159a053a" (UID: "3e1cce8d-f0cf-4f0e-ae15-1365159a053a"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 14:58:47.071743 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.071669    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e1cce8d-f0cf-4f0e-ae15-1365159a053a-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "3e1cce8d-f0cf-4f0e-ae15-1365159a053a" (UID: "3e1cce8d-f0cf-4f0e-ae15-1365159a053a"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 14:58:47.071743 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.071712    2560 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3e1cce8d-f0cf-4f0e-ae15-1365159a053a-metrics-client-ca\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Jul 23 14:58:47.072408 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.072357    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e1cce8d-f0cf-4f0e-ae15-1365159a053a-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "3e1cce8d-f0cf-4f0e-ae15-1365159a053a" (UID: "3e1cce8d-f0cf-4f0e-ae15-1365159a053a"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 14:58:47.075819 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.075758    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e1cce8d-f0cf-4f0e-ae15-1365159a053a-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "3e1cce8d-f0cf-4f0e-ae15-1365159a053a" (UID: "3e1cce8d-f0cf-4f0e-ae15-1365159a053a"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 14:58:47.076401 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.076355    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e1cce8d-f0cf-4f0e-ae15-1365159a053a-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "3e1cce8d-f0cf-4f0e-ae15-1365159a053a" (UID: "3e1cce8d-f0cf-4f0e-ae15-1365159a053a"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 14:58:47.076526 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.076472    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e1cce8d-f0cf-4f0e-ae15-1365159a053a-config-volume" (OuterVolumeSpecName: "config-volume") pod "3e1cce8d-f0cf-4f0e-ae15-1365159a053a" (UID: "3e1cce8d-f0cf-4f0e-ae15-1365159a053a"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 14:58:47.076641 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.076579    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e1cce8d-f0cf-4f0e-ae15-1365159a053a-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "3e1cce8d-f0cf-4f0e-ae15-1365159a053a" (UID: "3e1cce8d-f0cf-4f0e-ae15-1365159a053a"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 14:58:47.077186 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.077148    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e1cce8d-f0cf-4f0e-ae15-1365159a053a-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "3e1cce8d-f0cf-4f0e-ae15-1365159a053a" (UID: "3e1cce8d-f0cf-4f0e-ae15-1365159a053a"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 14:58:47.077860 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.077685    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e1cce8d-f0cf-4f0e-ae15-1365159a053a-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "3e1cce8d-f0cf-4f0e-ae15-1365159a053a" (UID: "3e1cce8d-f0cf-4f0e-ae15-1365159a053a"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 14:58:47.077860 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.077776    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e1cce8d-f0cf-4f0e-ae15-1365159a053a-config-out" (OuterVolumeSpecName: "config-out") pod "3e1cce8d-f0cf-4f0e-ae15-1365159a053a" (UID: "3e1cce8d-f0cf-4f0e-ae15-1365159a053a"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 14:58:47.077998 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.077822    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e1cce8d-f0cf-4f0e-ae15-1365159a053a-kube-api-access-qbs9x" (OuterVolumeSpecName: "kube-api-access-qbs9x") pod "3e1cce8d-f0cf-4f0e-ae15-1365159a053a" (UID: "3e1cce8d-f0cf-4f0e-ae15-1365159a053a"). InnerVolumeSpecName "kube-api-access-qbs9x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 14:58:47.082065 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.082043    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e1cce8d-f0cf-4f0e-ae15-1365159a053a-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "3e1cce8d-f0cf-4f0e-ae15-1365159a053a" (UID: "3e1cce8d-f0cf-4f0e-ae15-1365159a053a"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 14:58:47.089720 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.089687    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e1cce8d-f0cf-4f0e-ae15-1365159a053a-web-config" (OuterVolumeSpecName: "web-config") pod "3e1cce8d-f0cf-4f0e-ae15-1365159a053a" (UID: "3e1cce8d-f0cf-4f0e-ae15-1365159a053a"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 14:58:47.172975 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.172889    2560 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3e1cce8d-f0cf-4f0e-ae15-1365159a053a-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Jul 23 14:58:47.172975 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.172917    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbs9x\" (UniqueName: \"kubernetes.io/projected/3e1cce8d-f0cf-4f0e-ae15-1365159a053a-kube-api-access-qbs9x\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Jul 23 14:58:47.172975 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.172931    2560 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3e1cce8d-f0cf-4f0e-ae15-1365159a053a-secret-alertmanager-main-tls\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Jul 23 14:58:47.172975 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.172943    2560 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3e1cce8d-f0cf-4f0e-ae15-1365159a053a-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Jul 23 14:58:47.172975 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.172955    2560 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/3e1cce8d-f0cf-4f0e-ae15-1365159a053a-alertmanager-main-db\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Jul 23 14:58:47.172975 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.172966    2560 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3e1cce8d-f0cf-4f0e-ae15-1365159a053a-config-out\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Jul 23 14:58:47.172975 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.172980    2560 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3e1cce8d-f0cf-4f0e-ae15-1365159a053a-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Jul 23 14:58:47.173328 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.172992    2560 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3e1cce8d-f0cf-4f0e-ae15-1365159a053a-tls-assets\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Jul 23 14:58:47.173328 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.173003    2560 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3e1cce8d-f0cf-4f0e-ae15-1365159a053a-cluster-tls-config\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Jul 23 14:58:47.173328 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.173013    2560 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3e1cce8d-f0cf-4f0e-ae15-1365159a053a-web-config\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Jul 23 14:58:47.173328 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.173024    2560 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/3e1cce8d-f0cf-4f0e-ae15-1365159a053a-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Jul 23 14:58:47.173328 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.173036    2560 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3e1cce8d-f0cf-4f0e-ae15-1365159a053a-config-volume\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Jul 23 14:58:47.386190 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.386156    2560 generic.go:356] "Generic (PLEG): container finished" podID="3e1cce8d-f0cf-4f0e-ae15-1365159a053a" containerID="b64f86bef3a49309851ed097ac04545ab918535cd3ff4ed0ab2386becedf3cd4" exitCode=0
Jul 23 14:58:47.386190 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.386179    2560 generic.go:356] "Generic (PLEG): container finished" podID="3e1cce8d-f0cf-4f0e-ae15-1365159a053a" containerID="397bfb664ee74066daa96e163f418d3ea31901cfb3663cbb312dfffde499d025" exitCode=0
Jul 23 14:58:47.386470 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.386248    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 23 14:58:47.386470 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.386258    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3e1cce8d-f0cf-4f0e-ae15-1365159a053a","Type":"ContainerDied","Data":"b64f86bef3a49309851ed097ac04545ab918535cd3ff4ed0ab2386becedf3cd4"}
Jul 23 14:58:47.386470 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.386345    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3e1cce8d-f0cf-4f0e-ae15-1365159a053a","Type":"ContainerDied","Data":"397bfb664ee74066daa96e163f418d3ea31901cfb3663cbb312dfffde499d025"}
Jul 23 14:58:47.386470 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.386389    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3e1cce8d-f0cf-4f0e-ae15-1365159a053a","Type":"ContainerDied","Data":"bc7906f2c8012f2038f22e05049d9de909599e278251977230501c7f660bd16e"}
Jul 23 14:58:47.386470 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.386407    2560 scope.go:117] "RemoveContainer" containerID="816d67dfe01c9d1463bd704f5f3e7eb76d8a3c12dabdfb0f0a235826aff7da21"
Jul 23 14:58:47.394673 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.394648    2560 scope.go:117] "RemoveContainer" containerID="b64f86bef3a49309851ed097ac04545ab918535cd3ff4ed0ab2386becedf3cd4"
Jul 23 14:58:47.403833 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.402395    2560 scope.go:117] "RemoveContainer" containerID="db0501a851762d35ac6e4dde6c9402245ef142857a0d3eb876a513817e2a00ad"
Jul 23 14:58:47.411183 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.411155    2560 scope.go:117] "RemoveContainer" containerID="397bfb664ee74066daa96e163f418d3ea31901cfb3663cbb312dfffde499d025"
Jul 23 14:58:47.418090 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.418073    2560 scope.go:117] "RemoveContainer" containerID="2247b9bdb6272f07bba16a3fc55671f8d9812b11fc065b0e286fc581eb338127"
Jul 23 14:58:47.425333 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.425313    2560 scope.go:117] "RemoveContainer" containerID="c3856691cff2e572946b1dec6433343ed6ca9e88cb2b15d25568d7fbe38f0aa2"
Jul 23 14:58:47.432351 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.432316    2560 scope.go:117] "RemoveContainer" containerID="9a8f3465b23cd42e0d6ab40d06c9d912ab160210b2ebc175fd262cae044995ad"
Jul 23 14:58:47.437389 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.437366    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 23 14:58:47.440803 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.440785    2560 scope.go:117] "RemoveContainer" containerID="816d67dfe01c9d1463bd704f5f3e7eb76d8a3c12dabdfb0f0a235826aff7da21"
Jul 23 14:58:47.441114 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:58:47.441087    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"816d67dfe01c9d1463bd704f5f3e7eb76d8a3c12dabdfb0f0a235826aff7da21\": container with ID starting with 816d67dfe01c9d1463bd704f5f3e7eb76d8a3c12dabdfb0f0a235826aff7da21 not found: ID does not exist" containerID="816d67dfe01c9d1463bd704f5f3e7eb76d8a3c12dabdfb0f0a235826aff7da21"
Jul 23 14:58:47.441157 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.441122    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"816d67dfe01c9d1463bd704f5f3e7eb76d8a3c12dabdfb0f0a235826aff7da21"} err="failed to get container status \"816d67dfe01c9d1463bd704f5f3e7eb76d8a3c12dabdfb0f0a235826aff7da21\": rpc error: code = NotFound desc = could not find container \"816d67dfe01c9d1463bd704f5f3e7eb76d8a3c12dabdfb0f0a235826aff7da21\": container with ID starting with 816d67dfe01c9d1463bd704f5f3e7eb76d8a3c12dabdfb0f0a235826aff7da21 not found: ID does not exist"
Jul 23 14:58:47.441157 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.441153    2560 scope.go:117] "RemoveContainer" containerID="b64f86bef3a49309851ed097ac04545ab918535cd3ff4ed0ab2386becedf3cd4"
Jul 23 14:58:47.441420 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:58:47.441402    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b64f86bef3a49309851ed097ac04545ab918535cd3ff4ed0ab2386becedf3cd4\": container with ID starting with b64f86bef3a49309851ed097ac04545ab918535cd3ff4ed0ab2386becedf3cd4 not found: ID does not exist" containerID="b64f86bef3a49309851ed097ac04545ab918535cd3ff4ed0ab2386becedf3cd4"
Jul 23 14:58:47.441474 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.441425    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b64f86bef3a49309851ed097ac04545ab918535cd3ff4ed0ab2386becedf3cd4"} err="failed to get container status \"b64f86bef3a49309851ed097ac04545ab918535cd3ff4ed0ab2386becedf3cd4\": rpc error: code = NotFound desc = could not find container \"b64f86bef3a49309851ed097ac04545ab918535cd3ff4ed0ab2386becedf3cd4\": container with ID starting with b64f86bef3a49309851ed097ac04545ab918535cd3ff4ed0ab2386becedf3cd4 not found: ID does not exist"
Jul 23 14:58:47.441474 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.441441    2560 scope.go:117] "RemoveContainer" containerID="db0501a851762d35ac6e4dde6c9402245ef142857a0d3eb876a513817e2a00ad"
Jul 23 14:58:47.441662 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:58:47.441646    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db0501a851762d35ac6e4dde6c9402245ef142857a0d3eb876a513817e2a00ad\": container with ID starting with db0501a851762d35ac6e4dde6c9402245ef142857a0d3eb876a513817e2a00ad not found: ID does not exist" containerID="db0501a851762d35ac6e4dde6c9402245ef142857a0d3eb876a513817e2a00ad"
Jul 23 14:58:47.441698 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.441666    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db0501a851762d35ac6e4dde6c9402245ef142857a0d3eb876a513817e2a00ad"} err="failed to get container status \"db0501a851762d35ac6e4dde6c9402245ef142857a0d3eb876a513817e2a00ad\": rpc error: code = NotFound desc = could not find container \"db0501a851762d35ac6e4dde6c9402245ef142857a0d3eb876a513817e2a00ad\": container with ID starting with db0501a851762d35ac6e4dde6c9402245ef142857a0d3eb876a513817e2a00ad not found: ID does not exist"
Jul 23 14:58:47.441698 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.441681    2560 scope.go:117] "RemoveContainer" containerID="397bfb664ee74066daa96e163f418d3ea31901cfb3663cbb312dfffde499d025"
Jul 23 14:58:47.441873 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:58:47.441857    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"397bfb664ee74066daa96e163f418d3ea31901cfb3663cbb312dfffde499d025\": container with ID starting with 397bfb664ee74066daa96e163f418d3ea31901cfb3663cbb312dfffde499d025 not found: ID does not exist" containerID="397bfb664ee74066daa96e163f418d3ea31901cfb3663cbb312dfffde499d025"
Jul 23 14:58:47.441925 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.441874    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"397bfb664ee74066daa96e163f418d3ea31901cfb3663cbb312dfffde499d025"} err="failed to get container status \"397bfb664ee74066daa96e163f418d3ea31901cfb3663cbb312dfffde499d025\": rpc error: code = NotFound desc = could not find container \"397bfb664ee74066daa96e163f418d3ea31901cfb3663cbb312dfffde499d025\": container with ID starting with 397bfb664ee74066daa96e163f418d3ea31901cfb3663cbb312dfffde499d025 not found: ID does not exist"
Jul 23 14:58:47.441925 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.441885    2560 scope.go:117] "RemoveContainer" containerID="2247b9bdb6272f07bba16a3fc55671f8d9812b11fc065b0e286fc581eb338127"
Jul 23 14:58:47.442112 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:58:47.442098    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2247b9bdb6272f07bba16a3fc55671f8d9812b11fc065b0e286fc581eb338127\": container with ID starting with 2247b9bdb6272f07bba16a3fc55671f8d9812b11fc065b0e286fc581eb338127 not found: ID does not exist" containerID="2247b9bdb6272f07bba16a3fc55671f8d9812b11fc065b0e286fc581eb338127"
Jul 23 14:58:47.442154 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.442115    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2247b9bdb6272f07bba16a3fc55671f8d9812b11fc065b0e286fc581eb338127"} err="failed to get container status \"2247b9bdb6272f07bba16a3fc55671f8d9812b11fc065b0e286fc581eb338127\": rpc error: code = NotFound desc = could not find container \"2247b9bdb6272f07bba16a3fc55671f8d9812b11fc065b0e286fc581eb338127\": container with ID starting with 2247b9bdb6272f07bba16a3fc55671f8d9812b11fc065b0e286fc581eb338127 not found: ID does not exist"
Jul 23 14:58:47.442154 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.442125    2560 scope.go:117] "RemoveContainer" containerID="c3856691cff2e572946b1dec6433343ed6ca9e88cb2b15d25568d7fbe38f0aa2"
Jul 23 14:58:47.442405 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:58:47.442378    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3856691cff2e572946b1dec6433343ed6ca9e88cb2b15d25568d7fbe38f0aa2\": container with ID starting with c3856691cff2e572946b1dec6433343ed6ca9e88cb2b15d25568d7fbe38f0aa2 not found: ID does not exist" containerID="c3856691cff2e572946b1dec6433343ed6ca9e88cb2b15d25568d7fbe38f0aa2"
Jul 23 14:58:47.442494 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.442411    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3856691cff2e572946b1dec6433343ed6ca9e88cb2b15d25568d7fbe38f0aa2"} err="failed to get container status \"c3856691cff2e572946b1dec6433343ed6ca9e88cb2b15d25568d7fbe38f0aa2\": rpc error: code = NotFound desc = could not find container \"c3856691cff2e572946b1dec6433343ed6ca9e88cb2b15d25568d7fbe38f0aa2\": container with ID starting with c3856691cff2e572946b1dec6433343ed6ca9e88cb2b15d25568d7fbe38f0aa2 not found: ID does not exist"
Jul 23 14:58:47.442494 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.442426    2560 scope.go:117] "RemoveContainer" containerID="9a8f3465b23cd42e0d6ab40d06c9d912ab160210b2ebc175fd262cae044995ad"
Jul 23 14:58:47.442686 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:58:47.442670    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a8f3465b23cd42e0d6ab40d06c9d912ab160210b2ebc175fd262cae044995ad\": container with ID starting with 9a8f3465b23cd42e0d6ab40d06c9d912ab160210b2ebc175fd262cae044995ad not found: ID does not exist" containerID="9a8f3465b23cd42e0d6ab40d06c9d912ab160210b2ebc175fd262cae044995ad"
Jul 23 14:58:47.442724 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.442692    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a8f3465b23cd42e0d6ab40d06c9d912ab160210b2ebc175fd262cae044995ad"} err="failed to get container status \"9a8f3465b23cd42e0d6ab40d06c9d912ab160210b2ebc175fd262cae044995ad\": rpc error: code = NotFound desc = could not find container \"9a8f3465b23cd42e0d6ab40d06c9d912ab160210b2ebc175fd262cae044995ad\": container with ID starting with 9a8f3465b23cd42e0d6ab40d06c9d912ab160210b2ebc175fd262cae044995ad not found: ID does not exist"
Jul 23 14:58:47.442724 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.442707    2560 scope.go:117] "RemoveContainer" containerID="816d67dfe01c9d1463bd704f5f3e7eb76d8a3c12dabdfb0f0a235826aff7da21"
Jul 23 14:58:47.442907 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.442892    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"816d67dfe01c9d1463bd704f5f3e7eb76d8a3c12dabdfb0f0a235826aff7da21"} err="failed to get container status \"816d67dfe01c9d1463bd704f5f3e7eb76d8a3c12dabdfb0f0a235826aff7da21\": rpc error: code = NotFound desc = could not find container \"816d67dfe01c9d1463bd704f5f3e7eb76d8a3c12dabdfb0f0a235826aff7da21\": container with ID starting with 816d67dfe01c9d1463bd704f5f3e7eb76d8a3c12dabdfb0f0a235826aff7da21 not found: ID does not exist"
Jul 23 14:58:47.442947 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.442908    2560 scope.go:117] "RemoveContainer" containerID="b64f86bef3a49309851ed097ac04545ab918535cd3ff4ed0ab2386becedf3cd4"
Jul 23 14:58:47.443124 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.443106    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b64f86bef3a49309851ed097ac04545ab918535cd3ff4ed0ab2386becedf3cd4"} err="failed to get container status \"b64f86bef3a49309851ed097ac04545ab918535cd3ff4ed0ab2386becedf3cd4\": rpc error: code = NotFound desc = could not find container \"b64f86bef3a49309851ed097ac04545ab918535cd3ff4ed0ab2386becedf3cd4\": container with ID starting with b64f86bef3a49309851ed097ac04545ab918535cd3ff4ed0ab2386becedf3cd4 not found: ID does not exist"
Jul 23 14:58:47.443124 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.443123    2560 scope.go:117] "RemoveContainer" containerID="db0501a851762d35ac6e4dde6c9402245ef142857a0d3eb876a513817e2a00ad"
Jul 23 14:58:47.443342 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.443322    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db0501a851762d35ac6e4dde6c9402245ef142857a0d3eb876a513817e2a00ad"} err="failed to get container status \"db0501a851762d35ac6e4dde6c9402245ef142857a0d3eb876a513817e2a00ad\": rpc error: code = NotFound desc = could not find container \"db0501a851762d35ac6e4dde6c9402245ef142857a0d3eb876a513817e2a00ad\": container with ID starting with db0501a851762d35ac6e4dde6c9402245ef142857a0d3eb876a513817e2a00ad not found: ID does not exist"
Jul 23 14:58:47.443390 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.443342    2560 scope.go:117] "RemoveContainer" containerID="397bfb664ee74066daa96e163f418d3ea31901cfb3663cbb312dfffde499d025"
Jul 23 14:58:47.443556 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.443535    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"397bfb664ee74066daa96e163f418d3ea31901cfb3663cbb312dfffde499d025"} err="failed to get container status \"397bfb664ee74066daa96e163f418d3ea31901cfb3663cbb312dfffde499d025\": rpc error: code = NotFound desc = could not find container \"397bfb664ee74066daa96e163f418d3ea31901cfb3663cbb312dfffde499d025\": container with ID starting with 397bfb664ee74066daa96e163f418d3ea31901cfb3663cbb312dfffde499d025 not found: ID does not exist"
Jul 23 14:58:47.443601 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.443565    2560 scope.go:117] "RemoveContainer" containerID="2247b9bdb6272f07bba16a3fc55671f8d9812b11fc065b0e286fc581eb338127"
Jul 23 14:58:47.443782 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.443765    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2247b9bdb6272f07bba16a3fc55671f8d9812b11fc065b0e286fc581eb338127"} err="failed to get container status \"2247b9bdb6272f07bba16a3fc55671f8d9812b11fc065b0e286fc581eb338127\": rpc error: code = NotFound desc = could not find container \"2247b9bdb6272f07bba16a3fc55671f8d9812b11fc065b0e286fc581eb338127\": container with ID starting with 2247b9bdb6272f07bba16a3fc55671f8d9812b11fc065b0e286fc581eb338127 not found: ID does not exist"
Jul 23 14:58:47.443828 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.443782    2560 scope.go:117] "RemoveContainer" containerID="c3856691cff2e572946b1dec6433343ed6ca9e88cb2b15d25568d7fbe38f0aa2"
Jul 23 14:58:47.444003 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.443988    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3856691cff2e572946b1dec6433343ed6ca9e88cb2b15d25568d7fbe38f0aa2"} err="failed to get container status \"c3856691cff2e572946b1dec6433343ed6ca9e88cb2b15d25568d7fbe38f0aa2\": rpc error: code = NotFound desc = could not find container \"c3856691cff2e572946b1dec6433343ed6ca9e88cb2b15d25568d7fbe38f0aa2\": container with ID starting with c3856691cff2e572946b1dec6433343ed6ca9e88cb2b15d25568d7fbe38f0aa2 not found: ID does not exist"
Jul 23 14:58:47.444047 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.444004    2560 scope.go:117] "RemoveContainer" containerID="9a8f3465b23cd42e0d6ab40d06c9d912ab160210b2ebc175fd262cae044995ad"
Jul 23 14:58:47.444189 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.444172    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a8f3465b23cd42e0d6ab40d06c9d912ab160210b2ebc175fd262cae044995ad"} err="failed to get container status \"9a8f3465b23cd42e0d6ab40d06c9d912ab160210b2ebc175fd262cae044995ad\": rpc error: code = NotFound desc = could not find container \"9a8f3465b23cd42e0d6ab40d06c9d912ab160210b2ebc175fd262cae044995ad\": container with ID starting with 9a8f3465b23cd42e0d6ab40d06c9d912ab160210b2ebc175fd262cae044995ad not found: ID does not exist"
Jul 23 14:58:47.453708 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.453684    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 23 14:58:47.486854 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.486830    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 23 14:58:47.487091 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.487080    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a37889b3-f376-4734-b5d4-812cefd9c0fc" containerName="console"
Jul 23 14:58:47.487126 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.487093    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="a37889b3-f376-4734-b5d4-812cefd9c0fc" containerName="console"
Jul 23 14:58:47.487126 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.487101    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e1cce8d-f0cf-4f0e-ae15-1365159a053a" containerName="alertmanager"
Jul 23 14:58:47.487126 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.487107    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e1cce8d-f0cf-4f0e-ae15-1365159a053a" containerName="alertmanager"
Jul 23 14:58:47.487126 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.487113    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e1cce8d-f0cf-4f0e-ae15-1365159a053a" containerName="prom-label-proxy"
Jul 23 14:58:47.487126 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.487119    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e1cce8d-f0cf-4f0e-ae15-1365159a053a" containerName="prom-label-proxy"
Jul 23 14:58:47.487309 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.487133    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e1cce8d-f0cf-4f0e-ae15-1365159a053a" containerName="init-config-reloader"
Jul 23 14:58:47.487309 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.487140    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e1cce8d-f0cf-4f0e-ae15-1365159a053a" containerName="init-config-reloader"
Jul 23 14:58:47.487309 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.487146    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e1cce8d-f0cf-4f0e-ae15-1365159a053a" containerName="kube-rbac-proxy-metric"
Jul 23 14:58:47.487309 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.487150    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e1cce8d-f0cf-4f0e-ae15-1365159a053a" containerName="kube-rbac-proxy-metric"
Jul 23 14:58:47.487309 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.487157    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e1cce8d-f0cf-4f0e-ae15-1365159a053a" containerName="kube-rbac-proxy-web"
Jul 23 14:58:47.487309 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.487163    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e1cce8d-f0cf-4f0e-ae15-1365159a053a" containerName="kube-rbac-proxy-web"
Jul 23 14:58:47.487309 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.487174    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7df58b86-f921-4316-b8b1-f48137124a76" containerName="registry"
Jul 23 14:58:47.487309 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.487179    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="7df58b86-f921-4316-b8b1-f48137124a76" containerName="registry"
Jul 23 14:58:47.487309 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.487185    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e1cce8d-f0cf-4f0e-ae15-1365159a053a" containerName="config-reloader"
Jul 23 14:58:47.487309 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.487190    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e1cce8d-f0cf-4f0e-ae15-1365159a053a" containerName="config-reloader"
Jul 23 14:58:47.487309 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.487196    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e1cce8d-f0cf-4f0e-ae15-1365159a053a" containerName="kube-rbac-proxy"
Jul 23 14:58:47.487309 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.487201    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e1cce8d-f0cf-4f0e-ae15-1365159a053a" containerName="kube-rbac-proxy"
Jul 23 14:58:47.487309 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.487255    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="a37889b3-f376-4734-b5d4-812cefd9c0fc" containerName="console"
Jul 23 14:58:47.487309 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.487262    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e1cce8d-f0cf-4f0e-ae15-1365159a053a" containerName="alertmanager"
Jul 23 14:58:47.487309 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.487282    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e1cce8d-f0cf-4f0e-ae15-1365159a053a" containerName="kube-rbac-proxy"
Jul 23 14:58:47.487309 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.487289    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e1cce8d-f0cf-4f0e-ae15-1365159a053a" containerName="config-reloader"
Jul 23 14:58:47.487309 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.487296    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e1cce8d-f0cf-4f0e-ae15-1365159a053a" containerName="kube-rbac-proxy-web"
Jul 23 14:58:47.487309 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.487301    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e1cce8d-f0cf-4f0e-ae15-1365159a053a" containerName="prom-label-proxy"
Jul 23 14:58:47.487309 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.487307    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e1cce8d-f0cf-4f0e-ae15-1365159a053a" containerName="kube-rbac-proxy-metric"
Jul 23 14:58:47.487309 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.487313    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="7df58b86-f921-4316-b8b1-f48137124a76" containerName="registry"
Jul 23 14:58:47.491365 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.491349    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 23 14:58:47.495350 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.495331    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 23 14:58:47.495483 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.495354    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 23 14:58:47.495483 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.495331    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-6nz66\""
Jul 23 14:58:47.495589 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.495543    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 23 14:58:47.495675 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.495591    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 23 14:58:47.495675 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.495610    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 23 14:58:47.495675 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.495610    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 23 14:58:47.495824 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.495742    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 23 14:58:47.495824 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.495817    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 23 14:58:47.501182 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.501164    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 23 14:58:47.506272 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.506254    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 23 14:58:47.576557 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.576522    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/766a6fa8-2283-4a2f-9f94-9ccfeaa59434-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"766a6fa8-2283-4a2f-9f94-9ccfeaa59434\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 14:58:47.576557 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.576555    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/766a6fa8-2283-4a2f-9f94-9ccfeaa59434-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"766a6fa8-2283-4a2f-9f94-9ccfeaa59434\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 14:58:47.576773 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.576587    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/766a6fa8-2283-4a2f-9f94-9ccfeaa59434-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"766a6fa8-2283-4a2f-9f94-9ccfeaa59434\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 14:58:47.576773 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.576625    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/766a6fa8-2283-4a2f-9f94-9ccfeaa59434-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"766a6fa8-2283-4a2f-9f94-9ccfeaa59434\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 14:58:47.576773 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.576658    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/766a6fa8-2283-4a2f-9f94-9ccfeaa59434-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"766a6fa8-2283-4a2f-9f94-9ccfeaa59434\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 14:58:47.576773 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.576718    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/766a6fa8-2283-4a2f-9f94-9ccfeaa59434-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"766a6fa8-2283-4a2f-9f94-9ccfeaa59434\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 14:58:47.576773 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.576736    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/766a6fa8-2283-4a2f-9f94-9ccfeaa59434-tls-assets\") pod \"alertmanager-main-0\" (UID: \"766a6fa8-2283-4a2f-9f94-9ccfeaa59434\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 14:58:47.576961 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.576782    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/766a6fa8-2283-4a2f-9f94-9ccfeaa59434-config-volume\") pod \"alertmanager-main-0\" (UID: \"766a6fa8-2283-4a2f-9f94-9ccfeaa59434\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 14:58:47.576961 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.576811    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/766a6fa8-2283-4a2f-9f94-9ccfeaa59434-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"766a6fa8-2283-4a2f-9f94-9ccfeaa59434\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 14:58:47.576961 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.576826    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwbgf\" (UniqueName: \"kubernetes.io/projected/766a6fa8-2283-4a2f-9f94-9ccfeaa59434-kube-api-access-bwbgf\") pod \"alertmanager-main-0\" (UID: \"766a6fa8-2283-4a2f-9f94-9ccfeaa59434\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 14:58:47.576961 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.576861    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/766a6fa8-2283-4a2f-9f94-9ccfeaa59434-web-config\") pod \"alertmanager-main-0\" (UID: \"766a6fa8-2283-4a2f-9f94-9ccfeaa59434\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 14:58:47.576961 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.576881    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/766a6fa8-2283-4a2f-9f94-9ccfeaa59434-config-out\") pod \"alertmanager-main-0\" (UID: \"766a6fa8-2283-4a2f-9f94-9ccfeaa59434\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 14:58:47.576961 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.576901    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/766a6fa8-2283-4a2f-9f94-9ccfeaa59434-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"766a6fa8-2283-4a2f-9f94-9ccfeaa59434\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 14:58:47.677961 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.677874    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/766a6fa8-2283-4a2f-9f94-9ccfeaa59434-web-config\") pod \"alertmanager-main-0\" (UID: \"766a6fa8-2283-4a2f-9f94-9ccfeaa59434\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 14:58:47.677961 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.677905    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/766a6fa8-2283-4a2f-9f94-9ccfeaa59434-config-out\") pod \"alertmanager-main-0\" (UID: \"766a6fa8-2283-4a2f-9f94-9ccfeaa59434\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 14:58:47.677961 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.677926    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/766a6fa8-2283-4a2f-9f94-9ccfeaa59434-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"766a6fa8-2283-4a2f-9f94-9ccfeaa59434\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 14:58:47.677961 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.677947    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/766a6fa8-2283-4a2f-9f94-9ccfeaa59434-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"766a6fa8-2283-4a2f-9f94-9ccfeaa59434\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 14:58:47.678298 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.677969    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/766a6fa8-2283-4a2f-9f94-9ccfeaa59434-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"766a6fa8-2283-4a2f-9f94-9ccfeaa59434\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 14:58:47.678298 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.677999    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/766a6fa8-2283-4a2f-9f94-9ccfeaa59434-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"766a6fa8-2283-4a2f-9f94-9ccfeaa59434\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 14:58:47.678298 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.678022    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/766a6fa8-2283-4a2f-9f94-9ccfeaa59434-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"766a6fa8-2283-4a2f-9f94-9ccfeaa59434\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 14:58:47.678298 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.678051    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/766a6fa8-2283-4a2f-9f94-9ccfeaa59434-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"766a6fa8-2283-4a2f-9f94-9ccfeaa59434\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 14:58:47.678298 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.678116    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/766a6fa8-2283-4a2f-9f94-9ccfeaa59434-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"766a6fa8-2283-4a2f-9f94-9ccfeaa59434\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 14:58:47.678298 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.678152    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/766a6fa8-2283-4a2f-9f94-9ccfeaa59434-tls-assets\") pod \"alertmanager-main-0\" (UID: \"766a6fa8-2283-4a2f-9f94-9ccfeaa59434\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 14:58:47.678298 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.678175    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/766a6fa8-2283-4a2f-9f94-9ccfeaa59434-config-volume\") pod \"alertmanager-main-0\" (UID: \"766a6fa8-2283-4a2f-9f94-9ccfeaa59434\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 14:58:47.678298 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.678252    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/766a6fa8-2283-4a2f-9f94-9ccfeaa59434-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"766a6fa8-2283-4a2f-9f94-9ccfeaa59434\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 14:58:47.678298 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.678278    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwbgf\" (UniqueName: \"kubernetes.io/projected/766a6fa8-2283-4a2f-9f94-9ccfeaa59434-kube-api-access-bwbgf\") pod \"alertmanager-main-0\" (UID: \"766a6fa8-2283-4a2f-9f94-9ccfeaa59434\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 14:58:47.679437 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.678852    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/766a6fa8-2283-4a2f-9f94-9ccfeaa59434-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"766a6fa8-2283-4a2f-9f94-9ccfeaa59434\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 14:58:47.679437 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.679086    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/766a6fa8-2283-4a2f-9f94-9ccfeaa59434-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"766a6fa8-2283-4a2f-9f94-9ccfeaa59434\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 14:58:47.679630 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.679518    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/766a6fa8-2283-4a2f-9f94-9ccfeaa59434-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"766a6fa8-2283-4a2f-9f94-9ccfeaa59434\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 14:58:47.681295 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.681268    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/766a6fa8-2283-4a2f-9f94-9ccfeaa59434-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"766a6fa8-2283-4a2f-9f94-9ccfeaa59434\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 14:58:47.681414 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.681296    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/766a6fa8-2283-4a2f-9f94-9ccfeaa59434-config-out\") pod \"alertmanager-main-0\" (UID: \"766a6fa8-2283-4a2f-9f94-9ccfeaa59434\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 14:58:47.681414 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.681367    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/766a6fa8-2283-4a2f-9f94-9ccfeaa59434-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"766a6fa8-2283-4a2f-9f94-9ccfeaa59434\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 14:58:47.681538 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.681498    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/766a6fa8-2283-4a2f-9f94-9ccfeaa59434-config-volume\") pod \"alertmanager-main-0\" (UID: \"766a6fa8-2283-4a2f-9f94-9ccfeaa59434\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 14:58:47.681775 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.681756    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/766a6fa8-2283-4a2f-9f94-9ccfeaa59434-web-config\") pod \"alertmanager-main-0\" (UID: \"766a6fa8-2283-4a2f-9f94-9ccfeaa59434\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 14:58:47.681925 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.681910    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/766a6fa8-2283-4a2f-9f94-9ccfeaa59434-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"766a6fa8-2283-4a2f-9f94-9ccfeaa59434\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 14:58:47.682253 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.682205    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/766a6fa8-2283-4a2f-9f94-9ccfeaa59434-tls-assets\") pod \"alertmanager-main-0\" (UID: \"766a6fa8-2283-4a2f-9f94-9ccfeaa59434\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 14:58:47.682253 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.682214    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/766a6fa8-2283-4a2f-9f94-9ccfeaa59434-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"766a6fa8-2283-4a2f-9f94-9ccfeaa59434\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 14:58:47.682854 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.682837    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/766a6fa8-2283-4a2f-9f94-9ccfeaa59434-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"766a6fa8-2283-4a2f-9f94-9ccfeaa59434\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 14:58:47.689364 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.689345    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwbgf\" (UniqueName: \"kubernetes.io/projected/766a6fa8-2283-4a2f-9f94-9ccfeaa59434-kube-api-access-bwbgf\") pod \"alertmanager-main-0\" (UID: \"766a6fa8-2283-4a2f-9f94-9ccfeaa59434\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 14:58:47.800721 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.800684    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 23 14:58:47.936358 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:47.936252    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 23 14:58:47.938895 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:58:47.938869    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod766a6fa8_2283_4a2f_9f94_9ccfeaa59434.slice/crio-d3d4596254b2bd26f986ace1fe8242184d3e3722ee676beae581867f5ce3067d WatchSource:0}: Error finding container d3d4596254b2bd26f986ace1fe8242184d3e3722ee676beae581867f5ce3067d: Status 404 returned error can't find the container with id d3d4596254b2bd26f986ace1fe8242184d3e3722ee676beae581867f5ce3067d
Jul 23 14:58:48.391925 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:48.391895    2560 generic.go:356] "Generic (PLEG): container finished" podID="766a6fa8-2283-4a2f-9f94-9ccfeaa59434" containerID="224d2ca1db030f8f58437095f9566cc500d4d8faa969cac0d81932e0c405f717" exitCode=0
Jul 23 14:58:48.392389 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:48.391942    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"766a6fa8-2283-4a2f-9f94-9ccfeaa59434","Type":"ContainerDied","Data":"224d2ca1db030f8f58437095f9566cc500d4d8faa969cac0d81932e0c405f717"}
Jul 23 14:58:48.392389 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:48.391989    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"766a6fa8-2283-4a2f-9f94-9ccfeaa59434","Type":"ContainerStarted","Data":"d3d4596254b2bd26f986ace1fe8242184d3e3722ee676beae581867f5ce3067d"}
Jul 23 14:58:49.061898 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:49.061861    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3e1cce8d-f0cf-4f0e-ae15-1365159a053a" path="/var/lib/kubelet/pods/3e1cce8d-f0cf-4f0e-ae15-1365159a053a/volumes"
Jul 23 14:58:49.399021 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:49.398933    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"766a6fa8-2283-4a2f-9f94-9ccfeaa59434","Type":"ContainerStarted","Data":"8c4341a5fb9e0e2c71fde6fc0315db6b2b976bdc7b20a3ed1ab143a445d3b8ea"}
Jul 23 14:58:49.399021 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:49.399014    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"766a6fa8-2283-4a2f-9f94-9ccfeaa59434","Type":"ContainerStarted","Data":"f101767eac169665ebff9b9a4870919543308e6bfab4dcda38bf07f273863905"}
Jul 23 14:58:49.399427 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:49.399056    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"766a6fa8-2283-4a2f-9f94-9ccfeaa59434","Type":"ContainerStarted","Data":"8f4b4cc0afd479a1afa165727f9ea4605b842b38fa36cf4c2b3f10412f65f20c"}
Jul 23 14:58:49.399427 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:49.399096    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"766a6fa8-2283-4a2f-9f94-9ccfeaa59434","Type":"ContainerStarted","Data":"adb895a392b23e2e8fb7815f769c3da711ecc30417ffa45420f54f520f4835db"}
Jul 23 14:58:49.399427 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:49.399130    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"766a6fa8-2283-4a2f-9f94-9ccfeaa59434","Type":"ContainerStarted","Data":"39a8f39a9f2bbfa89718e48ec398186347f35478a858de17b71aa038d3c7f7ba"}
Jul 23 14:58:49.399427 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:49.399165    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"766a6fa8-2283-4a2f-9f94-9ccfeaa59434","Type":"ContainerStarted","Data":"6c2d077f5170c72b4d662ef4af5cc63f4990a9e4914493049ba9763c7389d0b8"}
Jul 23 14:58:49.429100 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:49.429046    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.414415363 podStartE2EDuration="2.429029477s" totalImagesPullingTime="0s" totalInitContainerRuntime="14.614114ms" isStatefulPod=false podCreationTimestamp="2026-07-23 14:58:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 14:58:49.426177272 +0000 UTC m=+370.934051032" watchObservedRunningTime="2026-07-23 14:58:49.429029477 +0000 UTC m=+370.936903278"
Jul 23 14:58:50.040182 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:50.040151    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 23 14:58:50.040485 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:50.040449    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="0e02c795-e710-498b-90e9-7a846f16504b" containerName="kube-rbac-proxy" containerID="cri-o://582a86367ba1729f8aa11b25481bab1dec45f569ba6dab4828cde9b5a69c48bb" gracePeriod=600
Jul 23 14:58:50.040485 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:50.040475    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="0e02c795-e710-498b-90e9-7a846f16504b" containerName="kube-rbac-proxy-thanos" containerID="cri-o://e85bdce164ebac8a98ec711f7add11f2fad6a75a9e923c57a536b8e9a8d82269" gracePeriod=600
Jul 23 14:58:50.040625 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:50.040483    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="0e02c795-e710-498b-90e9-7a846f16504b" containerName="kube-rbac-proxy-web" containerID="cri-o://5e2ae07470542d1ef4168a8d5182463c0c9bb4efa031ca33463962a9a5dbe9f8" gracePeriod=600
Jul 23 14:58:50.040625 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:50.040474    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="0e02c795-e710-498b-90e9-7a846f16504b" containerName="thanos-sidecar" containerID="cri-o://c1b2025ca2641246eb41918b12b99a699820606d0486afdc0387fc52768c0be6" gracePeriod=600
Jul 23 14:58:50.040625 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:50.040528    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="0e02c795-e710-498b-90e9-7a846f16504b" containerName="prometheus" containerID="cri-o://fb9a82dce1d1218395b7d51b4ceaf2dd7a35bcd1335bb1785db44de2a013baca" gracePeriod=600
Jul 23 14:58:50.040625 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:50.040450    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="0e02c795-e710-498b-90e9-7a846f16504b" containerName="config-reloader" containerID="cri-o://68b9f976fae08385ab6b37ea178f3abecde29cd40a180a0ee35e55caf2442a69" gracePeriod=600
Jul 23 14:58:50.407550 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:50.407462    2560 generic.go:356] "Generic (PLEG): container finished" podID="0e02c795-e710-498b-90e9-7a846f16504b" containerID="e85bdce164ebac8a98ec711f7add11f2fad6a75a9e923c57a536b8e9a8d82269" exitCode=0
Jul 23 14:58:50.407550 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:50.407489    2560 generic.go:356] "Generic (PLEG): container finished" podID="0e02c795-e710-498b-90e9-7a846f16504b" containerID="582a86367ba1729f8aa11b25481bab1dec45f569ba6dab4828cde9b5a69c48bb" exitCode=0
Jul 23 14:58:50.407550 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:50.407497    2560 generic.go:356] "Generic (PLEG): container finished" podID="0e02c795-e710-498b-90e9-7a846f16504b" containerID="c1b2025ca2641246eb41918b12b99a699820606d0486afdc0387fc52768c0be6" exitCode=0
Jul 23 14:58:50.407550 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:50.407503    2560 generic.go:356] "Generic (PLEG): container finished" podID="0e02c795-e710-498b-90e9-7a846f16504b" containerID="68b9f976fae08385ab6b37ea178f3abecde29cd40a180a0ee35e55caf2442a69" exitCode=0
Jul 23 14:58:50.407550 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:50.407508    2560 generic.go:356] "Generic (PLEG): container finished" podID="0e02c795-e710-498b-90e9-7a846f16504b" containerID="fb9a82dce1d1218395b7d51b4ceaf2dd7a35bcd1335bb1785db44de2a013baca" exitCode=0
Jul 23 14:58:50.407550 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:50.407530    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0e02c795-e710-498b-90e9-7a846f16504b","Type":"ContainerDied","Data":"e85bdce164ebac8a98ec711f7add11f2fad6a75a9e923c57a536b8e9a8d82269"}
Jul 23 14:58:50.408034 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:50.407602    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0e02c795-e710-498b-90e9-7a846f16504b","Type":"ContainerDied","Data":"582a86367ba1729f8aa11b25481bab1dec45f569ba6dab4828cde9b5a69c48bb"}
Jul 23 14:58:50.408034 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:50.407630    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0e02c795-e710-498b-90e9-7a846f16504b","Type":"ContainerDied","Data":"c1b2025ca2641246eb41918b12b99a699820606d0486afdc0387fc52768c0be6"}
Jul 23 14:58:50.408034 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:50.407654    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0e02c795-e710-498b-90e9-7a846f16504b","Type":"ContainerDied","Data":"68b9f976fae08385ab6b37ea178f3abecde29cd40a180a0ee35e55caf2442a69"}
Jul 23 14:58:50.408034 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:50.407676    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0e02c795-e710-498b-90e9-7a846f16504b","Type":"ContainerDied","Data":"fb9a82dce1d1218395b7d51b4ceaf2dd7a35bcd1335bb1785db44de2a013baca"}
Jul 23 14:58:51.283649 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.283627    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 14:58:51.309617 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.309581    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0e02c795-e710-498b-90e9-7a846f16504b-secret-grpc-tls\") pod \"0e02c795-e710-498b-90e9-7a846f16504b\" (UID: \"0e02c795-e710-498b-90e9-7a846f16504b\") "
Jul 23 14:58:51.309617 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.309615    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0e02c795-e710-498b-90e9-7a846f16504b-config-out\") pod \"0e02c795-e710-498b-90e9-7a846f16504b\" (UID: \"0e02c795-e710-498b-90e9-7a846f16504b\") "
Jul 23 14:58:51.309804 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.309632    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e02c795-e710-498b-90e9-7a846f16504b-prometheus-trusted-ca-bundle\") pod \"0e02c795-e710-498b-90e9-7a846f16504b\" (UID: \"0e02c795-e710-498b-90e9-7a846f16504b\") "
Jul 23 14:58:51.309804 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.309665    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0e02c795-e710-498b-90e9-7a846f16504b-prometheus-k8s-rulefiles-0\") pod \"0e02c795-e710-498b-90e9-7a846f16504b\" (UID: \"0e02c795-e710-498b-90e9-7a846f16504b\") "
Jul 23 14:58:51.309907 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.309884    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0e02c795-e710-498b-90e9-7a846f16504b-prometheus-k8s-rulefiles-2\") pod \"0e02c795-e710-498b-90e9-7a846f16504b\" (UID: \"0e02c795-e710-498b-90e9-7a846f16504b\") "
Jul 23 14:58:51.309964 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.309941    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0e02c795-e710-498b-90e9-7a846f16504b-thanos-prometheus-http-client-file\") pod \"0e02c795-e710-498b-90e9-7a846f16504b\" (UID: \"0e02c795-e710-498b-90e9-7a846f16504b\") "
Jul 23 14:58:51.310017 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.309979    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0e02c795-e710-498b-90e9-7a846f16504b-prometheus-k8s-rulefiles-1\") pod \"0e02c795-e710-498b-90e9-7a846f16504b\" (UID: \"0e02c795-e710-498b-90e9-7a846f16504b\") "
Jul 23 14:58:51.310017 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.310009    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e02c795-e710-498b-90e9-7a846f16504b-configmap-serving-certs-ca-bundle\") pod \"0e02c795-e710-498b-90e9-7a846f16504b\" (UID: \"0e02c795-e710-498b-90e9-7a846f16504b\") "
Jul 23 14:58:51.310111 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.310038    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0e02c795-e710-498b-90e9-7a846f16504b-prometheus-k8s-db\") pod \"0e02c795-e710-498b-90e9-7a846f16504b\" (UID: \"0e02c795-e710-498b-90e9-7a846f16504b\") "
Jul 23 14:58:51.310111 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.310069    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0e02c795-e710-498b-90e9-7a846f16504b-configmap-metrics-client-ca\") pod \"0e02c795-e710-498b-90e9-7a846f16504b\" (UID: \"0e02c795-e710-498b-90e9-7a846f16504b\") "
Jul 23 14:58:51.310111 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.310069    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e02c795-e710-498b-90e9-7a846f16504b-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "0e02c795-e710-498b-90e9-7a846f16504b" (UID: "0e02c795-e710-498b-90e9-7a846f16504b"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 14:58:51.310295 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.310113    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0e02c795-e710-498b-90e9-7a846f16504b-secret-prometheus-k8s-tls\") pod \"0e02c795-e710-498b-90e9-7a846f16504b\" (UID: \"0e02c795-e710-498b-90e9-7a846f16504b\") "
Jul 23 14:58:51.310295 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.310145    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0e02c795-e710-498b-90e9-7a846f16504b-web-config\") pod \"0e02c795-e710-498b-90e9-7a846f16504b\" (UID: \"0e02c795-e710-498b-90e9-7a846f16504b\") "
Jul 23 14:58:51.310295 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.310165    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/0e02c795-e710-498b-90e9-7a846f16504b-config\") pod \"0e02c795-e710-498b-90e9-7a846f16504b\" (UID: \"0e02c795-e710-498b-90e9-7a846f16504b\") "
Jul 23 14:58:51.310295 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.310187    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0e02c795-e710-498b-90e9-7a846f16504b-tls-assets\") pod \"0e02c795-e710-498b-90e9-7a846f16504b\" (UID: \"0e02c795-e710-498b-90e9-7a846f16504b\") "
Jul 23 14:58:51.310295 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.310220    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e02c795-e710-498b-90e9-7a846f16504b-configmap-kubelet-serving-ca-bundle\") pod \"0e02c795-e710-498b-90e9-7a846f16504b\" (UID: \"0e02c795-e710-498b-90e9-7a846f16504b\") "
Jul 23 14:58:51.310295 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.310268    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f68f6\" (UniqueName: \"kubernetes.io/projected/0e02c795-e710-498b-90e9-7a846f16504b-kube-api-access-f68f6\") pod \"0e02c795-e710-498b-90e9-7a846f16504b\" (UID: \"0e02c795-e710-498b-90e9-7a846f16504b\") "
Jul 23 14:58:51.310573 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.310303    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e02c795-e710-498b-90e9-7a846f16504b-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "0e02c795-e710-498b-90e9-7a846f16504b" (UID: "0e02c795-e710-498b-90e9-7a846f16504b"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 14:58:51.310573 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.310323    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0e02c795-e710-498b-90e9-7a846f16504b-secret-metrics-client-certs\") pod \"0e02c795-e710-498b-90e9-7a846f16504b\" (UID: \"0e02c795-e710-498b-90e9-7a846f16504b\") "
Jul 23 14:58:51.310573 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.310352    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0e02c795-e710-498b-90e9-7a846f16504b-secret-kube-rbac-proxy\") pod \"0e02c795-e710-498b-90e9-7a846f16504b\" (UID: \"0e02c795-e710-498b-90e9-7a846f16504b\") "
Jul 23 14:58:51.310573 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.310394    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e02c795-e710-498b-90e9-7a846f16504b-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "0e02c795-e710-498b-90e9-7a846f16504b" (UID: "0e02c795-e710-498b-90e9-7a846f16504b"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 14:58:51.310573 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.310400    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e02c795-e710-498b-90e9-7a846f16504b-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "0e02c795-e710-498b-90e9-7a846f16504b" (UID: "0e02c795-e710-498b-90e9-7a846f16504b"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 14:58:51.310573 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.310409    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0e02c795-e710-498b-90e9-7a846f16504b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"0e02c795-e710-498b-90e9-7a846f16504b\" (UID: \"0e02c795-e710-498b-90e9-7a846f16504b\") "
Jul 23 14:58:51.315567 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.312955    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e02c795-e710-498b-90e9-7a846f16504b-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "0e02c795-e710-498b-90e9-7a846f16504b" (UID: "0e02c795-e710-498b-90e9-7a846f16504b"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 14:58:51.315567 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.313349    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e02c795-e710-498b-90e9-7a846f16504b-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "0e02c795-e710-498b-90e9-7a846f16504b" (UID: "0e02c795-e710-498b-90e9-7a846f16504b"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 14:58:51.315567 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.313504    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e02c795-e710-498b-90e9-7a846f16504b-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "0e02c795-e710-498b-90e9-7a846f16504b" (UID: "0e02c795-e710-498b-90e9-7a846f16504b"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 14:58:51.315567 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.313600    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e02c795-e710-498b-90e9-7a846f16504b-config-out" (OuterVolumeSpecName: "config-out") pod "0e02c795-e710-498b-90e9-7a846f16504b" (UID: "0e02c795-e710-498b-90e9-7a846f16504b"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 14:58:51.315567 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.313943    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e02c795-e710-498b-90e9-7a846f16504b-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "0e02c795-e710-498b-90e9-7a846f16504b" (UID: "0e02c795-e710-498b-90e9-7a846f16504b"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 14:58:51.315567 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.315695    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e02c795-e710-498b-90e9-7a846f16504b-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "0e02c795-e710-498b-90e9-7a846f16504b" (UID: "0e02c795-e710-498b-90e9-7a846f16504b"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 14:58:51.315567 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.317311    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e02c795-e710-498b-90e9-7a846f16504b-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "0e02c795-e710-498b-90e9-7a846f16504b" (UID: "0e02c795-e710-498b-90e9-7a846f16504b"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 14:58:51.315567 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.317455    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e02c795-e710-498b-90e9-7a846f16504b-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "0e02c795-e710-498b-90e9-7a846f16504b" (UID: "0e02c795-e710-498b-90e9-7a846f16504b"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 14:58:51.315567 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.318280    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e02c795-e710-498b-90e9-7a846f16504b-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "0e02c795-e710-498b-90e9-7a846f16504b" (UID: "0e02c795-e710-498b-90e9-7a846f16504b"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 14:58:51.315567 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.318378    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0e02c795-e710-498b-90e9-7a846f16504b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"0e02c795-e710-498b-90e9-7a846f16504b\" (UID: \"0e02c795-e710-498b-90e9-7a846f16504b\") "
Jul 23 14:58:51.315567 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.321400    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e02c795-e710-498b-90e9-7a846f16504b-kube-api-access-f68f6" (OuterVolumeSpecName: "kube-api-access-f68f6") pod "0e02c795-e710-498b-90e9-7a846f16504b" (UID: "0e02c795-e710-498b-90e9-7a846f16504b"). InnerVolumeSpecName "kube-api-access-f68f6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 14:58:51.315567 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.321503    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e02c795-e710-498b-90e9-7a846f16504b-config" (OuterVolumeSpecName: "config") pod "0e02c795-e710-498b-90e9-7a846f16504b" (UID: "0e02c795-e710-498b-90e9-7a846f16504b"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 14:58:51.325042 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.324325    2560 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/0e02c795-e710-498b-90e9-7a846f16504b-config\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Jul 23 14:58:51.325042 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.324355    2560 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0e02c795-e710-498b-90e9-7a846f16504b-tls-assets\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Jul 23 14:58:51.325042 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.324381    2560 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e02c795-e710-498b-90e9-7a846f16504b-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Jul 23 14:58:51.325042 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.324399    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f68f6\" (UniqueName: \"kubernetes.io/projected/0e02c795-e710-498b-90e9-7a846f16504b-kube-api-access-f68f6\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Jul 23 14:58:51.325042 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.324415    2560 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0e02c795-e710-498b-90e9-7a846f16504b-secret-kube-rbac-proxy\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Jul 23 14:58:51.325042 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.324429    2560 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0e02c795-e710-498b-90e9-7a846f16504b-secret-grpc-tls\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Jul 23 14:58:51.325042 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.324447    2560 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0e02c795-e710-498b-90e9-7a846f16504b-config-out\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Jul 23 14:58:51.325042 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.324461    2560 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e02c795-e710-498b-90e9-7a846f16504b-prometheus-trusted-ca-bundle\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Jul 23 14:58:51.325042 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.324474    2560 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0e02c795-e710-498b-90e9-7a846f16504b-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Jul 23 14:58:51.325042 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.324488    2560 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0e02c795-e710-498b-90e9-7a846f16504b-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Jul 23 14:58:51.325042 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.324509    2560 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0e02c795-e710-498b-90e9-7a846f16504b-thanos-prometheus-http-client-file\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Jul 23 14:58:51.325042 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.324523    2560 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0e02c795-e710-498b-90e9-7a846f16504b-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Jul 23 14:58:51.325042 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.324552    2560 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e02c795-e710-498b-90e9-7a846f16504b-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Jul 23 14:58:51.325042 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.324566    2560 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0e02c795-e710-498b-90e9-7a846f16504b-prometheus-k8s-db\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Jul 23 14:58:51.325042 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.324584    2560 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0e02c795-e710-498b-90e9-7a846f16504b-configmap-metrics-client-ca\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Jul 23 14:58:51.327824 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.327749    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e02c795-e710-498b-90e9-7a846f16504b-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "0e02c795-e710-498b-90e9-7a846f16504b" (UID: "0e02c795-e710-498b-90e9-7a846f16504b"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 14:58:51.328287 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.327872    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e02c795-e710-498b-90e9-7a846f16504b-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "0e02c795-e710-498b-90e9-7a846f16504b" (UID: "0e02c795-e710-498b-90e9-7a846f16504b"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 14:58:51.328287 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.328034    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e02c795-e710-498b-90e9-7a846f16504b-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "0e02c795-e710-498b-90e9-7a846f16504b" (UID: "0e02c795-e710-498b-90e9-7a846f16504b"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 14:58:51.328287 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.328164    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e02c795-e710-498b-90e9-7a846f16504b-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "0e02c795-e710-498b-90e9-7a846f16504b" (UID: "0e02c795-e710-498b-90e9-7a846f16504b"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 14:58:51.338065 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.338043    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e02c795-e710-498b-90e9-7a846f16504b-web-config" (OuterVolumeSpecName: "web-config") pod "0e02c795-e710-498b-90e9-7a846f16504b" (UID: "0e02c795-e710-498b-90e9-7a846f16504b"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 14:58:51.414742 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.414649    2560 generic.go:356] "Generic (PLEG): container finished" podID="0e02c795-e710-498b-90e9-7a846f16504b" containerID="5e2ae07470542d1ef4168a8d5182463c0c9bb4efa031ca33463962a9a5dbe9f8" exitCode=0
Jul 23 14:58:51.415095 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.414742    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 14:58:51.415095 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.414735    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0e02c795-e710-498b-90e9-7a846f16504b","Type":"ContainerDied","Data":"5e2ae07470542d1ef4168a8d5182463c0c9bb4efa031ca33463962a9a5dbe9f8"}
Jul 23 14:58:51.415095 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.414906    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0e02c795-e710-498b-90e9-7a846f16504b","Type":"ContainerDied","Data":"2cc6f70b1c351703c3c59d998c231fbeb4311d82cf61cd78593a6e5587d5e662"}
Jul 23 14:58:51.415095 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.414943    2560 scope.go:117] "RemoveContainer" containerID="e85bdce164ebac8a98ec711f7add11f2fad6a75a9e923c57a536b8e9a8d82269"
Jul 23 14:58:51.423155 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.423138    2560 scope.go:117] "RemoveContainer" containerID="582a86367ba1729f8aa11b25481bab1dec45f569ba6dab4828cde9b5a69c48bb"
Jul 23 14:58:51.425329 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.425311    2560 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0e02c795-e710-498b-90e9-7a846f16504b-secret-prometheus-k8s-tls\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Jul 23 14:58:51.425329 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.425330    2560 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0e02c795-e710-498b-90e9-7a846f16504b-web-config\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Jul 23 14:58:51.425529 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.425341    2560 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0e02c795-e710-498b-90e9-7a846f16504b-secret-metrics-client-certs\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Jul 23 14:58:51.425529 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.425350    2560 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0e02c795-e710-498b-90e9-7a846f16504b-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Jul 23 14:58:51.425529 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.425360    2560 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0e02c795-e710-498b-90e9-7a846f16504b-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Jul 23 14:58:51.432198 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.430903    2560 scope.go:117] "RemoveContainer" containerID="5e2ae07470542d1ef4168a8d5182463c0c9bb4efa031ca33463962a9a5dbe9f8"
Jul 23 14:58:51.438610 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.438594    2560 scope.go:117] "RemoveContainer" containerID="c1b2025ca2641246eb41918b12b99a699820606d0486afdc0387fc52768c0be6"
Jul 23 14:58:51.439008 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.438987    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 23 14:58:51.445688 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.445656    2560 scope.go:117] "RemoveContainer" containerID="68b9f976fae08385ab6b37ea178f3abecde29cd40a180a0ee35e55caf2442a69"
Jul 23 14:58:51.447840 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.447823    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 23 14:58:51.452658 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.452632    2560 scope.go:117] "RemoveContainer" containerID="fb9a82dce1d1218395b7d51b4ceaf2dd7a35bcd1335bb1785db44de2a013baca"
Jul 23 14:58:51.459867 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.459846    2560 scope.go:117] "RemoveContainer" containerID="81d8a68383821ddfb67b911a924c49baadd716f0cc43e42e004673cae527de01"
Jul 23 14:58:51.467069 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.467052    2560 scope.go:117] "RemoveContainer" containerID="e85bdce164ebac8a98ec711f7add11f2fad6a75a9e923c57a536b8e9a8d82269"
Jul 23 14:58:51.467357 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:58:51.467338    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e85bdce164ebac8a98ec711f7add11f2fad6a75a9e923c57a536b8e9a8d82269\": container with ID starting with e85bdce164ebac8a98ec711f7add11f2fad6a75a9e923c57a536b8e9a8d82269 not found: ID does not exist" containerID="e85bdce164ebac8a98ec711f7add11f2fad6a75a9e923c57a536b8e9a8d82269"
Jul 23 14:58:51.467414 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.467367    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e85bdce164ebac8a98ec711f7add11f2fad6a75a9e923c57a536b8e9a8d82269"} err="failed to get container status \"e85bdce164ebac8a98ec711f7add11f2fad6a75a9e923c57a536b8e9a8d82269\": rpc error: code = NotFound desc = could not find container \"e85bdce164ebac8a98ec711f7add11f2fad6a75a9e923c57a536b8e9a8d82269\": container with ID starting with e85bdce164ebac8a98ec711f7add11f2fad6a75a9e923c57a536b8e9a8d82269 not found: ID does not exist"
Jul 23 14:58:51.467414 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.467387    2560 scope.go:117] "RemoveContainer" containerID="582a86367ba1729f8aa11b25481bab1dec45f569ba6dab4828cde9b5a69c48bb"
Jul 23 14:58:51.467647 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:58:51.467625    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"582a86367ba1729f8aa11b25481bab1dec45f569ba6dab4828cde9b5a69c48bb\": container with ID starting with 582a86367ba1729f8aa11b25481bab1dec45f569ba6dab4828cde9b5a69c48bb not found: ID does not exist" containerID="582a86367ba1729f8aa11b25481bab1dec45f569ba6dab4828cde9b5a69c48bb"
Jul 23 14:58:51.467691 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.467652    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"582a86367ba1729f8aa11b25481bab1dec45f569ba6dab4828cde9b5a69c48bb"} err="failed to get container status \"582a86367ba1729f8aa11b25481bab1dec45f569ba6dab4828cde9b5a69c48bb\": rpc error: code = NotFound desc = could not find container \"582a86367ba1729f8aa11b25481bab1dec45f569ba6dab4828cde9b5a69c48bb\": container with ID starting with 582a86367ba1729f8aa11b25481bab1dec45f569ba6dab4828cde9b5a69c48bb not found: ID does not exist"
Jul 23 14:58:51.467691 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.467669    2560 scope.go:117] "RemoveContainer" containerID="5e2ae07470542d1ef4168a8d5182463c0c9bb4efa031ca33463962a9a5dbe9f8"
Jul 23 14:58:51.467926 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:58:51.467909    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e2ae07470542d1ef4168a8d5182463c0c9bb4efa031ca33463962a9a5dbe9f8\": container with ID starting with 5e2ae07470542d1ef4168a8d5182463c0c9bb4efa031ca33463962a9a5dbe9f8 not found: ID does not exist" containerID="5e2ae07470542d1ef4168a8d5182463c0c9bb4efa031ca33463962a9a5dbe9f8"
Jul 23 14:58:51.467985 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.467934    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e2ae07470542d1ef4168a8d5182463c0c9bb4efa031ca33463962a9a5dbe9f8"} err="failed to get container status \"5e2ae07470542d1ef4168a8d5182463c0c9bb4efa031ca33463962a9a5dbe9f8\": rpc error: code = NotFound desc = could not find container \"5e2ae07470542d1ef4168a8d5182463c0c9bb4efa031ca33463962a9a5dbe9f8\": container with ID starting with 5e2ae07470542d1ef4168a8d5182463c0c9bb4efa031ca33463962a9a5dbe9f8 not found: ID does not exist"
Jul 23 14:58:51.467985 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.467957    2560 scope.go:117] "RemoveContainer" containerID="c1b2025ca2641246eb41918b12b99a699820606d0486afdc0387fc52768c0be6"
Jul 23 14:58:51.468203 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:58:51.468185    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1b2025ca2641246eb41918b12b99a699820606d0486afdc0387fc52768c0be6\": container with ID starting with c1b2025ca2641246eb41918b12b99a699820606d0486afdc0387fc52768c0be6 not found: ID does not exist" containerID="c1b2025ca2641246eb41918b12b99a699820606d0486afdc0387fc52768c0be6"
Jul 23 14:58:51.468275 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.468208    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1b2025ca2641246eb41918b12b99a699820606d0486afdc0387fc52768c0be6"} err="failed to get container status \"c1b2025ca2641246eb41918b12b99a699820606d0486afdc0387fc52768c0be6\": rpc error: code = NotFound desc = could not find container \"c1b2025ca2641246eb41918b12b99a699820606d0486afdc0387fc52768c0be6\": container with ID starting with c1b2025ca2641246eb41918b12b99a699820606d0486afdc0387fc52768c0be6 not found: ID does not exist"
Jul 23 14:58:51.468275 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.468250    2560 scope.go:117] "RemoveContainer" containerID="68b9f976fae08385ab6b37ea178f3abecde29cd40a180a0ee35e55caf2442a69"
Jul 23 14:58:51.468487 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:58:51.468471    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68b9f976fae08385ab6b37ea178f3abecde29cd40a180a0ee35e55caf2442a69\": container with ID starting with 68b9f976fae08385ab6b37ea178f3abecde29cd40a180a0ee35e55caf2442a69 not found: ID does not exist" containerID="68b9f976fae08385ab6b37ea178f3abecde29cd40a180a0ee35e55caf2442a69"
Jul 23 14:58:51.468541 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.468494    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68b9f976fae08385ab6b37ea178f3abecde29cd40a180a0ee35e55caf2442a69"} err="failed to get container status \"68b9f976fae08385ab6b37ea178f3abecde29cd40a180a0ee35e55caf2442a69\": rpc error: code = NotFound desc = could not find container \"68b9f976fae08385ab6b37ea178f3abecde29cd40a180a0ee35e55caf2442a69\": container with ID starting with 68b9f976fae08385ab6b37ea178f3abecde29cd40a180a0ee35e55caf2442a69 not found: ID does not exist"
Jul 23 14:58:51.468541 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.468514    2560 scope.go:117] "RemoveContainer" containerID="fb9a82dce1d1218395b7d51b4ceaf2dd7a35bcd1335bb1785db44de2a013baca"
Jul 23 14:58:51.468763 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:58:51.468746    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb9a82dce1d1218395b7d51b4ceaf2dd7a35bcd1335bb1785db44de2a013baca\": container with ID starting with fb9a82dce1d1218395b7d51b4ceaf2dd7a35bcd1335bb1785db44de2a013baca not found: ID does not exist" containerID="fb9a82dce1d1218395b7d51b4ceaf2dd7a35bcd1335bb1785db44de2a013baca"
Jul 23 14:58:51.468797 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.468768    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb9a82dce1d1218395b7d51b4ceaf2dd7a35bcd1335bb1785db44de2a013baca"} err="failed to get container status \"fb9a82dce1d1218395b7d51b4ceaf2dd7a35bcd1335bb1785db44de2a013baca\": rpc error: code = NotFound desc = could not find container \"fb9a82dce1d1218395b7d51b4ceaf2dd7a35bcd1335bb1785db44de2a013baca\": container with ID starting with fb9a82dce1d1218395b7d51b4ceaf2dd7a35bcd1335bb1785db44de2a013baca not found: ID does not exist"
Jul 23 14:58:51.468797 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.468783    2560 scope.go:117] "RemoveContainer" containerID="81d8a68383821ddfb67b911a924c49baadd716f0cc43e42e004673cae527de01"
Jul 23 14:58:51.468990 ip-10-0-141-105 kubenswrapper[2560]: E0723 14:58:51.468975    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81d8a68383821ddfb67b911a924c49baadd716f0cc43e42e004673cae527de01\": container with ID starting with 81d8a68383821ddfb67b911a924c49baadd716f0cc43e42e004673cae527de01 not found: ID does not exist" containerID="81d8a68383821ddfb67b911a924c49baadd716f0cc43e42e004673cae527de01"
Jul 23 14:58:51.469028 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.468995    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81d8a68383821ddfb67b911a924c49baadd716f0cc43e42e004673cae527de01"} err="failed to get container status \"81d8a68383821ddfb67b911a924c49baadd716f0cc43e42e004673cae527de01\": rpc error: code = NotFound desc = could not find container \"81d8a68383821ddfb67b911a924c49baadd716f0cc43e42e004673cae527de01\": container with ID starting with 81d8a68383821ddfb67b911a924c49baadd716f0cc43e42e004673cae527de01 not found: ID does not exist"
Jul 23 14:58:51.473581 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.473560    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 23 14:58:51.473868 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.473853    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e02c795-e710-498b-90e9-7a846f16504b" containerName="kube-rbac-proxy-web"
Jul 23 14:58:51.473932 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.473871    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e02c795-e710-498b-90e9-7a846f16504b" containerName="kube-rbac-proxy-web"
Jul 23 14:58:51.473932 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.473887    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e02c795-e710-498b-90e9-7a846f16504b" containerName="config-reloader"
Jul 23 14:58:51.473932 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.473898    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e02c795-e710-498b-90e9-7a846f16504b" containerName="config-reloader"
Jul 23 14:58:51.473932 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.473908    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e02c795-e710-498b-90e9-7a846f16504b" containerName="thanos-sidecar"
Jul 23 14:58:51.473932 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.473916    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e02c795-e710-498b-90e9-7a846f16504b" containerName="thanos-sidecar"
Jul 23 14:58:51.474186 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.473940    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e02c795-e710-498b-90e9-7a846f16504b" containerName="kube-rbac-proxy"
Jul 23 14:58:51.474186 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.473948    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e02c795-e710-498b-90e9-7a846f16504b" containerName="kube-rbac-proxy"
Jul 23 14:58:51.474186 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.473958    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e02c795-e710-498b-90e9-7a846f16504b" containerName="init-config-reloader"
Jul 23 14:58:51.474186 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.473967    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e02c795-e710-498b-90e9-7a846f16504b" containerName="init-config-reloader"
Jul 23 14:58:51.474186 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.473980    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e02c795-e710-498b-90e9-7a846f16504b" containerName="kube-rbac-proxy-thanos"
Jul 23 14:58:51.474186 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.473990    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e02c795-e710-498b-90e9-7a846f16504b" containerName="kube-rbac-proxy-thanos"
Jul 23 14:58:51.474186 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.474005    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e02c795-e710-498b-90e9-7a846f16504b" containerName="prometheus"
Jul 23 14:58:51.474186 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.474014    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e02c795-e710-498b-90e9-7a846f16504b" containerName="prometheus"
Jul 23 14:58:51.474186 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.474080    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e02c795-e710-498b-90e9-7a846f16504b" containerName="kube-rbac-proxy"
Jul 23 14:58:51.474186 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.474093    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e02c795-e710-498b-90e9-7a846f16504b" containerName="thanos-sidecar"
Jul 23 14:58:51.474186 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.474102    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e02c795-e710-498b-90e9-7a846f16504b" containerName="prometheus"
Jul 23 14:58:51.474186 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.474110    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e02c795-e710-498b-90e9-7a846f16504b" containerName="config-reloader"
Jul 23 14:58:51.474186 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.474120    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e02c795-e710-498b-90e9-7a846f16504b" containerName="kube-rbac-proxy-web"
Jul 23 14:58:51.474186 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.474131    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e02c795-e710-498b-90e9-7a846f16504b" containerName="kube-rbac-proxy-thanos"
Jul 23 14:58:51.478511 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.478494    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 14:58:51.481872 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.481854    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 23 14:58:51.481872 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.481842    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 23 14:58:51.482043 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.481955    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-qfqg2\""
Jul 23 14:58:51.482043 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.481961    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-6rr9scg9k0sk7\""
Jul 23 14:58:51.482043 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.482026    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 23 14:58:51.482043 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.482038    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 23 14:58:51.482349 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.482111    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 23 14:58:51.482349 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.482156    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 23 14:58:51.482467 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.482386    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 23 14:58:51.483161 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.483121    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 23 14:58:51.483161 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.483127    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 23 14:58:51.483377 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.483340    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 23 14:58:51.483377 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.483358    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 23 14:58:51.483581 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.483382    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 23 14:58:51.483581 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.483442    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 23 14:58:51.486667 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.486649    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 23 14:58:51.487411 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.487398    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 23 14:58:51.492088 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.492067    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 23 14:58:51.526630 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.526594    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/da9c516e-f152-474a-a04a-bfbdabaeb022-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"da9c516e-f152-474a-a04a-bfbdabaeb022\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 14:58:51.526630 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.526634    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da9c516e-f152-474a-a04a-bfbdabaeb022-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"da9c516e-f152-474a-a04a-bfbdabaeb022\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 14:58:51.526827 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.526659    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/da9c516e-f152-474a-a04a-bfbdabaeb022-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"da9c516e-f152-474a-a04a-bfbdabaeb022\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 14:58:51.526827 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.526726    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da9c516e-f152-474a-a04a-bfbdabaeb022-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"da9c516e-f152-474a-a04a-bfbdabaeb022\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 14:58:51.526827 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.526768    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/da9c516e-f152-474a-a04a-bfbdabaeb022-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"da9c516e-f152-474a-a04a-bfbdabaeb022\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 14:58:51.526827 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.526786    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/da9c516e-f152-474a-a04a-bfbdabaeb022-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"da9c516e-f152-474a-a04a-bfbdabaeb022\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 14:58:51.526827 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.526803    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/da9c516e-f152-474a-a04a-bfbdabaeb022-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"da9c516e-f152-474a-a04a-bfbdabaeb022\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 14:58:51.526827 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.526818    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ncb6\" (UniqueName: \"kubernetes.io/projected/da9c516e-f152-474a-a04a-bfbdabaeb022-kube-api-access-8ncb6\") pod \"prometheus-k8s-0\" (UID: \"da9c516e-f152-474a-a04a-bfbdabaeb022\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 14:58:51.527007 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.526838    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/da9c516e-f152-474a-a04a-bfbdabaeb022-config-out\") pod \"prometheus-k8s-0\" (UID: \"da9c516e-f152-474a-a04a-bfbdabaeb022\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 14:58:51.527007 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.526884    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/da9c516e-f152-474a-a04a-bfbdabaeb022-config\") pod \"prometheus-k8s-0\" (UID: \"da9c516e-f152-474a-a04a-bfbdabaeb022\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 14:58:51.527007 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.526900    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/da9c516e-f152-474a-a04a-bfbdabaeb022-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"da9c516e-f152-474a-a04a-bfbdabaeb022\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 14:58:51.527007 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.526964    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/da9c516e-f152-474a-a04a-bfbdabaeb022-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"da9c516e-f152-474a-a04a-bfbdabaeb022\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 14:58:51.527007 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.526981    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/da9c516e-f152-474a-a04a-bfbdabaeb022-web-config\") pod \"prometheus-k8s-0\" (UID: \"da9c516e-f152-474a-a04a-bfbdabaeb022\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 14:58:51.527147 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.527031    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/da9c516e-f152-474a-a04a-bfbdabaeb022-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"da9c516e-f152-474a-a04a-bfbdabaeb022\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 14:58:51.527147 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.527052    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/da9c516e-f152-474a-a04a-bfbdabaeb022-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"da9c516e-f152-474a-a04a-bfbdabaeb022\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 14:58:51.527147 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.527083    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/da9c516e-f152-474a-a04a-bfbdabaeb022-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"da9c516e-f152-474a-a04a-bfbdabaeb022\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 14:58:51.527147 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.527106    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/da9c516e-f152-474a-a04a-bfbdabaeb022-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"da9c516e-f152-474a-a04a-bfbdabaeb022\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 14:58:51.527147 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.527128    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/da9c516e-f152-474a-a04a-bfbdabaeb022-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"da9c516e-f152-474a-a04a-bfbdabaeb022\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 14:58:51.527317 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.527151    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da9c516e-f152-474a-a04a-bfbdabaeb022-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"da9c516e-f152-474a-a04a-bfbdabaeb022\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 14:58:51.527317 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.527170    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/da9c516e-f152-474a-a04a-bfbdabaeb022-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"da9c516e-f152-474a-a04a-bfbdabaeb022\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 14:58:51.628431 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.628386    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/da9c516e-f152-474a-a04a-bfbdabaeb022-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"da9c516e-f152-474a-a04a-bfbdabaeb022\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 14:58:51.628643 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.628459    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/da9c516e-f152-474a-a04a-bfbdabaeb022-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"da9c516e-f152-474a-a04a-bfbdabaeb022\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 14:58:51.628643 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.628476    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/da9c516e-f152-474a-a04a-bfbdabaeb022-web-config\") pod \"prometheus-k8s-0\" (UID: \"da9c516e-f152-474a-a04a-bfbdabaeb022\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 14:58:51.628643 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.628501    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/da9c516e-f152-474a-a04a-bfbdabaeb022-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"da9c516e-f152-474a-a04a-bfbdabaeb022\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 14:58:51.628643 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.628519    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/da9c516e-f152-474a-a04a-bfbdabaeb022-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"da9c516e-f152-474a-a04a-bfbdabaeb022\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 14:58:51.628643 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.628539    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/da9c516e-f152-474a-a04a-bfbdabaeb022-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"da9c516e-f152-474a-a04a-bfbdabaeb022\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 14:58:51.628643 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.628558    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/da9c516e-f152-474a-a04a-bfbdabaeb022-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"da9c516e-f152-474a-a04a-bfbdabaeb022\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 14:58:51.628643 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.628578    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/da9c516e-f152-474a-a04a-bfbdabaeb022-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"da9c516e-f152-474a-a04a-bfbdabaeb022\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 14:58:51.628643 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.628595    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da9c516e-f152-474a-a04a-bfbdabaeb022-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"da9c516e-f152-474a-a04a-bfbdabaeb022\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 14:58:51.628643 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.628622    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/da9c516e-f152-474a-a04a-bfbdabaeb022-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"da9c516e-f152-474a-a04a-bfbdabaeb022\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 14:58:51.629063 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.628656    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/da9c516e-f152-474a-a04a-bfbdabaeb022-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"da9c516e-f152-474a-a04a-bfbdabaeb022\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 14:58:51.629063 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.628678    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da9c516e-f152-474a-a04a-bfbdabaeb022-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"da9c516e-f152-474a-a04a-bfbdabaeb022\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 14:58:51.629063 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.628710    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/da9c516e-f152-474a-a04a-bfbdabaeb022-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"da9c516e-f152-474a-a04a-bfbdabaeb022\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 14:58:51.629063 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.628735    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da9c516e-f152-474a-a04a-bfbdabaeb022-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"da9c516e-f152-474a-a04a-bfbdabaeb022\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 14:58:51.629063 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.628759    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/da9c516e-f152-474a-a04a-bfbdabaeb022-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"da9c516e-f152-474a-a04a-bfbdabaeb022\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 14:58:51.629063 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.628780    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/da9c516e-f152-474a-a04a-bfbdabaeb022-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"da9c516e-f152-474a-a04a-bfbdabaeb022\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 14:58:51.629063 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.628803    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/da9c516e-f152-474a-a04a-bfbdabaeb022-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"da9c516e-f152-474a-a04a-bfbdabaeb022\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 14:58:51.629063 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.628825    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ncb6\" (UniqueName: \"kubernetes.io/projected/da9c516e-f152-474a-a04a-bfbdabaeb022-kube-api-access-8ncb6\") pod \"prometheus-k8s-0\" (UID: \"da9c516e-f152-474a-a04a-bfbdabaeb022\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 14:58:51.629063 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.628853    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/da9c516e-f152-474a-a04a-bfbdabaeb022-config-out\") pod \"prometheus-k8s-0\" (UID: \"da9c516e-f152-474a-a04a-bfbdabaeb022\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 14:58:51.629063 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.628883    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/da9c516e-f152-474a-a04a-bfbdabaeb022-config\") pod \"prometheus-k8s-0\" (UID: \"da9c516e-f152-474a-a04a-bfbdabaeb022\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 14:58:51.629606 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.629111    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/da9c516e-f152-474a-a04a-bfbdabaeb022-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"da9c516e-f152-474a-a04a-bfbdabaeb022\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 14:58:51.629941 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.629909    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da9c516e-f152-474a-a04a-bfbdabaeb022-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"da9c516e-f152-474a-a04a-bfbdabaeb022\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 14:58:51.630825 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.630795    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da9c516e-f152-474a-a04a-bfbdabaeb022-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"da9c516e-f152-474a-a04a-bfbdabaeb022\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 14:58:51.632163 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.631914    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/da9c516e-f152-474a-a04a-bfbdabaeb022-web-config\") pod \"prometheus-k8s-0\" (UID: \"da9c516e-f152-474a-a04a-bfbdabaeb022\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 14:58:51.632163 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.631927    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/da9c516e-f152-474a-a04a-bfbdabaeb022-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"da9c516e-f152-474a-a04a-bfbdabaeb022\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 14:58:51.632163 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.631941    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/da9c516e-f152-474a-a04a-bfbdabaeb022-config\") pod \"prometheus-k8s-0\" (UID: \"da9c516e-f152-474a-a04a-bfbdabaeb022\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 14:58:51.632163 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.632064    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/da9c516e-f152-474a-a04a-bfbdabaeb022-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"da9c516e-f152-474a-a04a-bfbdabaeb022\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 14:58:51.632163 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.632110    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/da9c516e-f152-474a-a04a-bfbdabaeb022-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"da9c516e-f152-474a-a04a-bfbdabaeb022\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 14:58:51.632673 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.632174    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/da9c516e-f152-474a-a04a-bfbdabaeb022-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"da9c516e-f152-474a-a04a-bfbdabaeb022\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 14:58:51.632673 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.632247    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/da9c516e-f152-474a-a04a-bfbdabaeb022-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"da9c516e-f152-474a-a04a-bfbdabaeb022\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 14:58:51.632673 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.632424    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/da9c516e-f152-474a-a04a-bfbdabaeb022-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"da9c516e-f152-474a-a04a-bfbdabaeb022\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 14:58:51.632835 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.632756    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/da9c516e-f152-474a-a04a-bfbdabaeb022-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"da9c516e-f152-474a-a04a-bfbdabaeb022\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 14:58:51.632835 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.632825    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/da9c516e-f152-474a-a04a-bfbdabaeb022-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"da9c516e-f152-474a-a04a-bfbdabaeb022\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 14:58:51.632932 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.632896    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da9c516e-f152-474a-a04a-bfbdabaeb022-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"da9c516e-f152-474a-a04a-bfbdabaeb022\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 14:58:51.633467 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.633442    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/da9c516e-f152-474a-a04a-bfbdabaeb022-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"da9c516e-f152-474a-a04a-bfbdabaeb022\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 14:58:51.634531 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.634505    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/da9c516e-f152-474a-a04a-bfbdabaeb022-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"da9c516e-f152-474a-a04a-bfbdabaeb022\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 14:58:51.634595 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.634580    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/da9c516e-f152-474a-a04a-bfbdabaeb022-config-out\") pod \"prometheus-k8s-0\" (UID: \"da9c516e-f152-474a-a04a-bfbdabaeb022\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 14:58:51.634715 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.634697    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/da9c516e-f152-474a-a04a-bfbdabaeb022-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"da9c516e-f152-474a-a04a-bfbdabaeb022\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 14:58:51.635100 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.635083    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/da9c516e-f152-474a-a04a-bfbdabaeb022-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"da9c516e-f152-474a-a04a-bfbdabaeb022\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 14:58:51.641744 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.641726    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ncb6\" (UniqueName: \"kubernetes.io/projected/da9c516e-f152-474a-a04a-bfbdabaeb022-kube-api-access-8ncb6\") pod \"prometheus-k8s-0\" (UID: \"da9c516e-f152-474a-a04a-bfbdabaeb022\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 14:58:51.788251 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.788195    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 14:58:51.934411 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:51.934381    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 23 14:58:51.937628 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:58:51.937601    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda9c516e_f152_474a_a04a_bfbdabaeb022.slice/crio-9326b01af1e2d500dacf373db44367f2bfcfa8eb43d82138476a020b77650ff3 WatchSource:0}: Error finding container 9326b01af1e2d500dacf373db44367f2bfcfa8eb43d82138476a020b77650ff3: Status 404 returned error can't find the container with id 9326b01af1e2d500dacf373db44367f2bfcfa8eb43d82138476a020b77650ff3
Jul 23 14:58:52.421137 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:52.421050    2560 generic.go:356] "Generic (PLEG): container finished" podID="da9c516e-f152-474a-a04a-bfbdabaeb022" containerID="74742275ea87bf4dea7ff43f7bdf3d2d103470f825770126c97ad7711d719ad0" exitCode=0
Jul 23 14:58:52.421137 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:52.421101    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"da9c516e-f152-474a-a04a-bfbdabaeb022","Type":"ContainerDied","Data":"74742275ea87bf4dea7ff43f7bdf3d2d103470f825770126c97ad7711d719ad0"}
Jul 23 14:58:52.421655 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:52.421150    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"da9c516e-f152-474a-a04a-bfbdabaeb022","Type":"ContainerStarted","Data":"9326b01af1e2d500dacf373db44367f2bfcfa8eb43d82138476a020b77650ff3"}
Jul 23 14:58:52.840882 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:52.840838    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/413d2464-7258-4246-85bb-ac9f03976ee5-metrics-certs\") pod \"network-metrics-daemon-hvw8v\" (UID: \"413d2464-7258-4246-85bb-ac9f03976ee5\") " pod="openshift-multus/network-metrics-daemon-hvw8v"
Jul 23 14:58:52.843674 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:52.843643    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/413d2464-7258-4246-85bb-ac9f03976ee5-metrics-certs\") pod \"network-metrics-daemon-hvw8v\" (UID: \"413d2464-7258-4246-85bb-ac9f03976ee5\") " pod="openshift-multus/network-metrics-daemon-hvw8v"
Jul 23 14:58:52.961835 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:52.961754    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-brw86\""
Jul 23 14:58:52.968690 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:52.968665    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hvw8v"
Jul 23 14:58:53.062000 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:53.061968    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e02c795-e710-498b-90e9-7a846f16504b" path="/var/lib/kubelet/pods/0e02c795-e710-498b-90e9-7a846f16504b/volumes"
Jul 23 14:58:53.095193 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:53.095163    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-hvw8v"]
Jul 23 14:58:53.097980 ip-10-0-141-105 kubenswrapper[2560]: W0723 14:58:53.097955    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod413d2464_7258_4246_85bb_ac9f03976ee5.slice/crio-866b7806689544613d828c0630ce77da6fba9f6cd1e05f8e74ab8a844840d590 WatchSource:0}: Error finding container 866b7806689544613d828c0630ce77da6fba9f6cd1e05f8e74ab8a844840d590: Status 404 returned error can't find the container with id 866b7806689544613d828c0630ce77da6fba9f6cd1e05f8e74ab8a844840d590
Jul 23 14:58:53.430503 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:53.430468    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"da9c516e-f152-474a-a04a-bfbdabaeb022","Type":"ContainerStarted","Data":"3e3f159c3b86c031d0301f2639c3f42a9c15cfd88946fefa389fd8e98fce4b73"}
Jul 23 14:58:53.430958 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:53.430553    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"da9c516e-f152-474a-a04a-bfbdabaeb022","Type":"ContainerStarted","Data":"0d1aa0accda4f6e7e6750bc8e5514d2bbe5adf1149cc6f7fbd7ecafc0f97ff8c"}
Jul 23 14:58:53.430958 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:53.430582    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"da9c516e-f152-474a-a04a-bfbdabaeb022","Type":"ContainerStarted","Data":"cdc9fa2fc33cd746d47aa88d828b10b392014d87502a73fb66d47687c3563d35"}
Jul 23 14:58:53.430958 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:53.430610    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"da9c516e-f152-474a-a04a-bfbdabaeb022","Type":"ContainerStarted","Data":"5da98c6220a47206ed077ee4b66dbbb42702687f32acf4b93f2326ecfd5ec331"}
Jul 23 14:58:53.430958 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:53.430645    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"da9c516e-f152-474a-a04a-bfbdabaeb022","Type":"ContainerStarted","Data":"b2f8f3b9cb50508825a27160ebef2514c6904bba985fe336da8ab11add017001"}
Jul 23 14:58:53.430958 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:53.430680    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"da9c516e-f152-474a-a04a-bfbdabaeb022","Type":"ContainerStarted","Data":"d51f185b401e48df6e37422ef6a4cdfb3c5de3b71beeceb511b818d10c286a8e"}
Jul 23 14:58:53.431687 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:53.431665    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hvw8v" event={"ID":"413d2464-7258-4246-85bb-ac9f03976ee5","Type":"ContainerStarted","Data":"866b7806689544613d828c0630ce77da6fba9f6cd1e05f8e74ab8a844840d590"}
Jul 23 14:58:53.461644 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:53.461598    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.441260338 podStartE2EDuration="2.461576731s" totalImagesPullingTime="0s" totalInitContainerRuntime="20.316393ms" isStatefulPod=false podCreationTimestamp="2026-07-23 14:58:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 14:58:53.460169263 +0000 UTC m=+374.968043009" watchObservedRunningTime="2026-07-23 14:58:53.461576731 +0000 UTC m=+374.969450482"
Jul 23 14:58:54.438343 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:54.438251    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hvw8v" event={"ID":"413d2464-7258-4246-85bb-ac9f03976ee5","Type":"ContainerStarted","Data":"b1fe618afbdeea565ac251969defe1e8df87e05e6621a911a4a7d49c62c13df4"}
Jul 23 14:58:54.438343 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:54.438313    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hvw8v" event={"ID":"413d2464-7258-4246-85bb-ac9f03976ee5","Type":"ContainerStarted","Data":"9686cd14440e79b1ad762b02bc07d8ef081274025cfd2a1dc75af59d22e4a547"}
Jul 23 14:58:54.455386 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:54.455331    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-hvw8v" podStartSLOduration=374.54218603 podStartE2EDuration="6m15.455314575s" totalImagesPullingTime="913.128545ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 14:52:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 14:58:54.45407942 +0000 UTC m=+375.961953170" watchObservedRunningTime="2026-07-23 14:58:54.455314575 +0000 UTC m=+375.963188323"
Jul 23 14:58:56.789045 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:58:56.789004    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 14:59:51.788869 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:59:51.788831    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 14:59:51.804778 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:59:51.804752    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 14:59:52.673019 ip-10-0-141-105 kubenswrapper[2560]: I0723 14:59:52.672989    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 15:01:12.556243 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:01:12.556197    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-56b97c94c-nb24s"]
Jul 23 15:01:12.559138 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:01:12.559120    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-56b97c94c-nb24s"
Jul 23 15:01:12.562454 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:01:12.562431    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 23 15:01:12.562583 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:01:12.562463    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-hqkq8\""
Jul 23 15:01:12.562583 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:01:12.562480    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 23 15:01:12.562583 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:01:12.562520    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 23 15:01:12.562716 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:01:12.562582    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 23 15:01:12.573508 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:01:12.573488    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-56b97c94c-nb24s"]
Jul 23 15:01:12.621749 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:01:12.621728    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-5857c6d56c-kqj6p"]
Jul 23 15:01:12.624636 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:01:12.624622    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5857c6d56c-kqj6p"
Jul 23 15:01:12.627686 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:01:12.627668    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 23 15:01:12.627834 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:01:12.627788    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 23 15:01:12.627911 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:01:12.627895    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 23 15:01:12.628146 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:01:12.628133    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 23 15:01:12.628399 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:01:12.628384    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 23 15:01:12.628589 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:01:12.628573    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-ttqtl\""
Jul 23 15:01:12.643702 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:01:12.643678    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5857c6d56c-kqj6p"]
Jul 23 15:01:12.685990 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:01:12.685969    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szxv7\" (UniqueName: \"kubernetes.io/projected/0090c8f7-34af-4caa-a4ad-6cb6d6152380-kube-api-access-szxv7\") pod \"opendatahub-operator-controller-manager-56b97c94c-nb24s\" (UID: \"0090c8f7-34af-4caa-a4ad-6cb6d6152380\") " pod="opendatahub/opendatahub-operator-controller-manager-56b97c94c-nb24s"
Jul 23 15:01:12.686097 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:01:12.685997    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0090c8f7-34af-4caa-a4ad-6cb6d6152380-webhook-cert\") pod \"opendatahub-operator-controller-manager-56b97c94c-nb24s\" (UID: \"0090c8f7-34af-4caa-a4ad-6cb6d6152380\") " pod="opendatahub/opendatahub-operator-controller-manager-56b97c94c-nb24s"
Jul 23 15:01:12.686097 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:01:12.686021    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0090c8f7-34af-4caa-a4ad-6cb6d6152380-tmp\") pod \"opendatahub-operator-controller-manager-56b97c94c-nb24s\" (UID: \"0090c8f7-34af-4caa-a4ad-6cb6d6152380\") " pod="opendatahub/opendatahub-operator-controller-manager-56b97c94c-nb24s"
Jul 23 15:01:12.686097 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:01:12.686091    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0090c8f7-34af-4caa-a4ad-6cb6d6152380-apiservice-cert\") pod \"opendatahub-operator-controller-manager-56b97c94c-nb24s\" (UID: \"0090c8f7-34af-4caa-a4ad-6cb6d6152380\") " pod="opendatahub/opendatahub-operator-controller-manager-56b97c94c-nb24s"
Jul 23 15:01:12.686195 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:01:12.686125    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0090c8f7-34af-4caa-a4ad-6cb6d6152380-manifests\") pod \"opendatahub-operator-controller-manager-56b97c94c-nb24s\" (UID: \"0090c8f7-34af-4caa-a4ad-6cb6d6152380\") " pod="opendatahub/opendatahub-operator-controller-manager-56b97c94c-nb24s"
Jul 23 15:01:12.786830 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:01:12.786804    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szxv7\" (UniqueName: \"kubernetes.io/projected/0090c8f7-34af-4caa-a4ad-6cb6d6152380-kube-api-access-szxv7\") pod \"opendatahub-operator-controller-manager-56b97c94c-nb24s\" (UID: \"0090c8f7-34af-4caa-a4ad-6cb6d6152380\") " pod="opendatahub/opendatahub-operator-controller-manager-56b97c94c-nb24s"
Jul 23 15:01:12.786943 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:01:12.786834    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/e38c06a3-ef37-4429-8198-72de28028b71-manager-config\") pod \"lws-controller-manager-5857c6d56c-kqj6p\" (UID: \"e38c06a3-ef37-4429-8198-72de28028b71\") " pod="openshift-lws-operator/lws-controller-manager-5857c6d56c-kqj6p"
Jul 23 15:01:12.786943 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:01:12.786851    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0090c8f7-34af-4caa-a4ad-6cb6d6152380-webhook-cert\") pod \"opendatahub-operator-controller-manager-56b97c94c-nb24s\" (UID: \"0090c8f7-34af-4caa-a4ad-6cb6d6152380\") " pod="opendatahub/opendatahub-operator-controller-manager-56b97c94c-nb24s"
Jul 23 15:01:12.786943 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:01:12.786869    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0090c8f7-34af-4caa-a4ad-6cb6d6152380-tmp\") pod \"opendatahub-operator-controller-manager-56b97c94c-nb24s\" (UID: \"0090c8f7-34af-4caa-a4ad-6cb6d6152380\") " pod="opendatahub/opendatahub-operator-controller-manager-56b97c94c-nb24s"
Jul 23 15:01:12.786943 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:01:12.786899    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e38c06a3-ef37-4429-8198-72de28028b71-metrics-cert\") pod \"lws-controller-manager-5857c6d56c-kqj6p\" (UID: \"e38c06a3-ef37-4429-8198-72de28028b71\") " pod="openshift-lws-operator/lws-controller-manager-5857c6d56c-kqj6p"
Jul 23 15:01:12.786943 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:01:12.786922    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dv4zh\" (UniqueName: \"kubernetes.io/projected/e38c06a3-ef37-4429-8198-72de28028b71-kube-api-access-dv4zh\") pod \"lws-controller-manager-5857c6d56c-kqj6p\" (UID: \"e38c06a3-ef37-4429-8198-72de28028b71\") " pod="openshift-lws-operator/lws-controller-manager-5857c6d56c-kqj6p"
Jul 23 15:01:12.787203 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:01:12.786954    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0090c8f7-34af-4caa-a4ad-6cb6d6152380-apiservice-cert\") pod \"opendatahub-operator-controller-manager-56b97c94c-nb24s\" (UID: \"0090c8f7-34af-4caa-a4ad-6cb6d6152380\") " pod="opendatahub/opendatahub-operator-controller-manager-56b97c94c-nb24s"
Jul 23 15:01:12.787203 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:01:12.786983    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0090c8f7-34af-4caa-a4ad-6cb6d6152380-manifests\") pod \"opendatahub-operator-controller-manager-56b97c94c-nb24s\" (UID: \"0090c8f7-34af-4caa-a4ad-6cb6d6152380\") " pod="opendatahub/opendatahub-operator-controller-manager-56b97c94c-nb24s"
Jul 23 15:01:12.787203 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:01:12.787054    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e38c06a3-ef37-4429-8198-72de28028b71-cert\") pod \"lws-controller-manager-5857c6d56c-kqj6p\" (UID: \"e38c06a3-ef37-4429-8198-72de28028b71\") " pod="openshift-lws-operator/lws-controller-manager-5857c6d56c-kqj6p"
Jul 23 15:01:12.787382 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:01:12.787343    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0090c8f7-34af-4caa-a4ad-6cb6d6152380-manifests\") pod \"opendatahub-operator-controller-manager-56b97c94c-nb24s\" (UID: \"0090c8f7-34af-4caa-a4ad-6cb6d6152380\") " pod="opendatahub/opendatahub-operator-controller-manager-56b97c94c-nb24s"
Jul 23 15:01:12.789202 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:01:12.789177    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0090c8f7-34af-4caa-a4ad-6cb6d6152380-tmp\") pod \"opendatahub-operator-controller-manager-56b97c94c-nb24s\" (UID: \"0090c8f7-34af-4caa-a4ad-6cb6d6152380\") " pod="opendatahub/opendatahub-operator-controller-manager-56b97c94c-nb24s"
Jul 23 15:01:12.789345 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:01:12.789322    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0090c8f7-34af-4caa-a4ad-6cb6d6152380-webhook-cert\") pod \"opendatahub-operator-controller-manager-56b97c94c-nb24s\" (UID: \"0090c8f7-34af-4caa-a4ad-6cb6d6152380\") " pod="opendatahub/opendatahub-operator-controller-manager-56b97c94c-nb24s"
Jul 23 15:01:12.789517 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:01:12.789499    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0090c8f7-34af-4caa-a4ad-6cb6d6152380-apiservice-cert\") pod \"opendatahub-operator-controller-manager-56b97c94c-nb24s\" (UID: \"0090c8f7-34af-4caa-a4ad-6cb6d6152380\") " pod="opendatahub/opendatahub-operator-controller-manager-56b97c94c-nb24s"
Jul 23 15:01:12.824089 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:01:12.824037    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-szxv7\" (UniqueName: \"kubernetes.io/projected/0090c8f7-34af-4caa-a4ad-6cb6d6152380-kube-api-access-szxv7\") pod \"opendatahub-operator-controller-manager-56b97c94c-nb24s\" (UID: \"0090c8f7-34af-4caa-a4ad-6cb6d6152380\") " pod="opendatahub/opendatahub-operator-controller-manager-56b97c94c-nb24s"
Jul 23 15:01:12.869031 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:01:12.869006    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-56b97c94c-nb24s"
Jul 23 15:01:12.888154 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:01:12.888127    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/e38c06a3-ef37-4429-8198-72de28028b71-manager-config\") pod \"lws-controller-manager-5857c6d56c-kqj6p\" (UID: \"e38c06a3-ef37-4429-8198-72de28028b71\") " pod="openshift-lws-operator/lws-controller-manager-5857c6d56c-kqj6p"
Jul 23 15:01:12.888294 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:01:12.888173    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e38c06a3-ef37-4429-8198-72de28028b71-metrics-cert\") pod \"lws-controller-manager-5857c6d56c-kqj6p\" (UID: \"e38c06a3-ef37-4429-8198-72de28028b71\") " pod="openshift-lws-operator/lws-controller-manager-5857c6d56c-kqj6p"
Jul 23 15:01:12.888294 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:01:12.888195    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dv4zh\" (UniqueName: \"kubernetes.io/projected/e38c06a3-ef37-4429-8198-72de28028b71-kube-api-access-dv4zh\") pod \"lws-controller-manager-5857c6d56c-kqj6p\" (UID: \"e38c06a3-ef37-4429-8198-72de28028b71\") " pod="openshift-lws-operator/lws-controller-manager-5857c6d56c-kqj6p"
Jul 23 15:01:12.888294 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:01:12.888275    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e38c06a3-ef37-4429-8198-72de28028b71-cert\") pod \"lws-controller-manager-5857c6d56c-kqj6p\" (UID: \"e38c06a3-ef37-4429-8198-72de28028b71\") " pod="openshift-lws-operator/lws-controller-manager-5857c6d56c-kqj6p"
Jul 23 15:01:12.888873 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:01:12.888850    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/e38c06a3-ef37-4429-8198-72de28028b71-manager-config\") pod \"lws-controller-manager-5857c6d56c-kqj6p\" (UID: \"e38c06a3-ef37-4429-8198-72de28028b71\") " pod="openshift-lws-operator/lws-controller-manager-5857c6d56c-kqj6p"
Jul 23 15:01:12.890832 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:01:12.890802    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e38c06a3-ef37-4429-8198-72de28028b71-cert\") pod \"lws-controller-manager-5857c6d56c-kqj6p\" (UID: \"e38c06a3-ef37-4429-8198-72de28028b71\") " pod="openshift-lws-operator/lws-controller-manager-5857c6d56c-kqj6p"
Jul 23 15:01:12.890915 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:01:12.890807    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e38c06a3-ef37-4429-8198-72de28028b71-metrics-cert\") pod \"lws-controller-manager-5857c6d56c-kqj6p\" (UID: \"e38c06a3-ef37-4429-8198-72de28028b71\") " pod="openshift-lws-operator/lws-controller-manager-5857c6d56c-kqj6p"
Jul 23 15:01:12.902804 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:01:12.902775    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dv4zh\" (UniqueName: \"kubernetes.io/projected/e38c06a3-ef37-4429-8198-72de28028b71-kube-api-access-dv4zh\") pod \"lws-controller-manager-5857c6d56c-kqj6p\" (UID: \"e38c06a3-ef37-4429-8198-72de28028b71\") " pod="openshift-lws-operator/lws-controller-manager-5857c6d56c-kqj6p"
Jul 23 15:01:12.933554 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:01:12.933525    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5857c6d56c-kqj6p"
Jul 23 15:01:13.033483 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:01:13.033436    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-56b97c94c-nb24s"]
Jul 23 15:01:13.036933 ip-10-0-141-105 kubenswrapper[2560]: W0723 15:01:13.036905    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0090c8f7_34af_4caa_a4ad_6cb6d6152380.slice/crio-69ae7919acc31cb3104d360025de4689d508ea255677560dc42fc8f077141c26 WatchSource:0}: Error finding container 69ae7919acc31cb3104d360025de4689d508ea255677560dc42fc8f077141c26: Status 404 returned error can't find the container with id 69ae7919acc31cb3104d360025de4689d508ea255677560dc42fc8f077141c26
Jul 23 15:01:13.079736 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:01:13.079651    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5857c6d56c-kqj6p"]
Jul 23 15:01:13.082497 ip-10-0-141-105 kubenswrapper[2560]: W0723 15:01:13.082473    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode38c06a3_ef37_4429_8198_72de28028b71.slice/crio-da3ed7d5b0faec94e3eae8e7004f569c8d96a2fb00f6b7f0c69ebd9fc74ba941 WatchSource:0}: Error finding container da3ed7d5b0faec94e3eae8e7004f569c8d96a2fb00f6b7f0c69ebd9fc74ba941: Status 404 returned error can't find the container with id da3ed7d5b0faec94e3eae8e7004f569c8d96a2fb00f6b7f0c69ebd9fc74ba941
Jul 23 15:01:13.973070 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:01:13.972995    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-56b97c94c-nb24s" event={"ID":"0090c8f7-34af-4caa-a4ad-6cb6d6152380","Type":"ContainerStarted","Data":"69ae7919acc31cb3104d360025de4689d508ea255677560dc42fc8f077141c26"}
Jul 23 15:01:13.974524 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:01:13.974494    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5857c6d56c-kqj6p" event={"ID":"e38c06a3-ef37-4429-8198-72de28028b71","Type":"ContainerStarted","Data":"da3ed7d5b0faec94e3eae8e7004f569c8d96a2fb00f6b7f0c69ebd9fc74ba941"}
Jul 23 15:01:17.993505 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:01:17.993462    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-56b97c94c-nb24s" event={"ID":"0090c8f7-34af-4caa-a4ad-6cb6d6152380","Type":"ContainerStarted","Data":"6a91716a69f7b86529886891f1746f6b2739cc427dcbe45123f3e3d51b89aeaf"}
Jul 23 15:01:17.994908 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:01:17.994883    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5857c6d56c-kqj6p" event={"ID":"e38c06a3-ef37-4429-8198-72de28028b71","Type":"ContainerStarted","Data":"646f46e1787c89a37a5c4c096f802086a1985e2784fc14af2c348859309017d9"}
Jul 23 15:01:17.995051 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:01:17.995014    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-5857c6d56c-kqj6p"
Jul 23 15:01:18.043317 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:01:18.043259    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-5857c6d56c-kqj6p" podStartSLOduration=1.8404161650000002 podStartE2EDuration="6.04322601s" totalImagesPullingTime="4.202809845s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 15:01:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 15:01:18.039480227 +0000 UTC m=+519.547353971" watchObservedRunningTime="2026-07-23 15:01:18.04322601 +0000 UTC m=+519.551099756"
Jul 23 15:01:19.000545 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:01:19.000511    2560 generic.go:356] "Generic (PLEG): container finished" podID="0090c8f7-34af-4caa-a4ad-6cb6d6152380" containerID="6a91716a69f7b86529886891f1746f6b2739cc427dcbe45123f3e3d51b89aeaf" exitCode=0
Jul 23 15:01:19.000980 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:01:19.000603    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-56b97c94c-nb24s" event={"ID":"0090c8f7-34af-4caa-a4ad-6cb6d6152380","Type":"ContainerDied","Data":"6a91716a69f7b86529886891f1746f6b2739cc427dcbe45123f3e3d51b89aeaf"}
Jul 23 15:01:20.009855 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:01:20.009819    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-56b97c94c-nb24s" event={"ID":"0090c8f7-34af-4caa-a4ad-6cb6d6152380","Type":"ContainerStarted","Data":"8497f27917cef57dfbf5aa3825b74ba716840916c5f9f5b1d24c6a88bf185799"}
Jul 23 15:01:20.010181 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:01:20.009932    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-56b97c94c-nb24s"
Jul 23 15:01:20.034954 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:01:20.034898    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-56b97c94c-nb24s" podStartSLOduration=1.8563722440000001 podStartE2EDuration="8.034883316s" totalImagesPullingTime="5.104427003s" totalInitContainerRuntime="1.074084069s" isStatefulPod=false podCreationTimestamp="2026-07-23 15:01:12 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 15:01:20.03331956 +0000 UTC m=+521.541193306" watchObservedRunningTime="2026-07-23 15:01:20.034883316 +0000 UTC m=+521.542757064"
Jul 23 15:01:29.003084 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:01:29.003051    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-5857c6d56c-kqj6p"
Jul 23 15:01:31.016481 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:01:31.016450    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-56b97c94c-nb24s"
Jul 23 15:01:33.798216 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:01:33.798183    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-8d89f64f-gx5q6"]
Jul 23 15:01:33.800559 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:01:33.800539    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-8d89f64f-gx5q6"
Jul 23 15:01:33.804724 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:01:33.804698    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-m76xf\""
Jul 23 15:01:33.806101 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:01:33.806083    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 23 15:01:33.806212 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:01:33.806167    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 23 15:01:33.806212 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:01:33.806128    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 23 15:01:33.806351 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:01:33.806328    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Jul 23 15:01:33.817076 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:01:33.817053    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-8d89f64f-gx5q6"]
Jul 23 15:01:33.868148 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:01:33.868123    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e08125f5-7cbf-4bf3-a666-56b514d5b964-tmp\") pod \"kube-auth-proxy-8d89f64f-gx5q6\" (UID: \"e08125f5-7cbf-4bf3-a666-56b514d5b964\") " pod="openshift-ingress/kube-auth-proxy-8d89f64f-gx5q6"
Jul 23 15:01:33.868284 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:01:33.868168    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mk7pt\" (UniqueName: \"kubernetes.io/projected/e08125f5-7cbf-4bf3-a666-56b514d5b964-kube-api-access-mk7pt\") pod \"kube-auth-proxy-8d89f64f-gx5q6\" (UID: \"e08125f5-7cbf-4bf3-a666-56b514d5b964\") " pod="openshift-ingress/kube-auth-proxy-8d89f64f-gx5q6"
Jul 23 15:01:33.868284 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:01:33.868200    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e08125f5-7cbf-4bf3-a666-56b514d5b964-tls-certs\") pod \"kube-auth-proxy-8d89f64f-gx5q6\" (UID: \"e08125f5-7cbf-4bf3-a666-56b514d5b964\") " pod="openshift-ingress/kube-auth-proxy-8d89f64f-gx5q6"
Jul 23 15:01:33.969495 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:01:33.969445    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e08125f5-7cbf-4bf3-a666-56b514d5b964-tmp\") pod \"kube-auth-proxy-8d89f64f-gx5q6\" (UID: \"e08125f5-7cbf-4bf3-a666-56b514d5b964\") " pod="openshift-ingress/kube-auth-proxy-8d89f64f-gx5q6"
Jul 23 15:01:33.969656 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:01:33.969522    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mk7pt\" (UniqueName: \"kubernetes.io/projected/e08125f5-7cbf-4bf3-a666-56b514d5b964-kube-api-access-mk7pt\") pod \"kube-auth-proxy-8d89f64f-gx5q6\" (UID: \"e08125f5-7cbf-4bf3-a666-56b514d5b964\") " pod="openshift-ingress/kube-auth-proxy-8d89f64f-gx5q6"
Jul 23 15:01:33.969656 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:01:33.969555    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e08125f5-7cbf-4bf3-a666-56b514d5b964-tls-certs\") pod \"kube-auth-proxy-8d89f64f-gx5q6\" (UID: \"e08125f5-7cbf-4bf3-a666-56b514d5b964\") " pod="openshift-ingress/kube-auth-proxy-8d89f64f-gx5q6"
Jul 23 15:01:33.971860 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:01:33.971829    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e08125f5-7cbf-4bf3-a666-56b514d5b964-tmp\") pod \"kube-auth-proxy-8d89f64f-gx5q6\" (UID: \"e08125f5-7cbf-4bf3-a666-56b514d5b964\") " pod="openshift-ingress/kube-auth-proxy-8d89f64f-gx5q6"
Jul 23 15:01:33.972054 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:01:33.972034    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e08125f5-7cbf-4bf3-a666-56b514d5b964-tls-certs\") pod \"kube-auth-proxy-8d89f64f-gx5q6\" (UID: \"e08125f5-7cbf-4bf3-a666-56b514d5b964\") " pod="openshift-ingress/kube-auth-proxy-8d89f64f-gx5q6"
Jul 23 15:01:33.992187 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:01:33.992158    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mk7pt\" (UniqueName: \"kubernetes.io/projected/e08125f5-7cbf-4bf3-a666-56b514d5b964-kube-api-access-mk7pt\") pod \"kube-auth-proxy-8d89f64f-gx5q6\" (UID: \"e08125f5-7cbf-4bf3-a666-56b514d5b964\") " pod="openshift-ingress/kube-auth-proxy-8d89f64f-gx5q6"
Jul 23 15:01:34.110891 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:01:34.110799    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-8d89f64f-gx5q6"
Jul 23 15:01:34.241359 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:01:34.241331    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-8d89f64f-gx5q6"]
Jul 23 15:01:34.243950 ip-10-0-141-105 kubenswrapper[2560]: W0723 15:01:34.243923    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode08125f5_7cbf_4bf3_a666_56b514d5b964.slice/crio-d335c34420a8fef477551d03c203e5a58dba11a5005f871cf28fcd5db6cd5d0d WatchSource:0}: Error finding container d335c34420a8fef477551d03c203e5a58dba11a5005f871cf28fcd5db6cd5d0d: Status 404 returned error can't find the container with id d335c34420a8fef477551d03c203e5a58dba11a5005f871cf28fcd5db6cd5d0d
Jul 23 15:01:35.070782 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:01:35.070743    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-8d89f64f-gx5q6" event={"ID":"e08125f5-7cbf-4bf3-a666-56b514d5b964","Type":"ContainerStarted","Data":"d335c34420a8fef477551d03c203e5a58dba11a5005f871cf28fcd5db6cd5d0d"}
Jul 23 15:01:37.081494 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:01:37.081394    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-8d89f64f-gx5q6" event={"ID":"e08125f5-7cbf-4bf3-a666-56b514d5b964","Type":"ContainerStarted","Data":"4732ea313814d1824c5aea5f9a72aa921bc386e2c6a8dbcf51231ca8a885840b"}
Jul 23 15:02:40.132518 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:02:40.132394    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z4fvn_e0c66f26-06fe-4273-ba29-fa20ce89b873/ovn-acl-logging/0.log"
Jul 23 15:02:40.134684 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:02:40.133973    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z4fvn_e0c66f26-06fe-4273-ba29-fa20ce89b873/ovn-acl-logging/0.log"
Jul 23 15:03:29.829993 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:03:29.829936    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-8d89f64f-gx5q6" podStartSLOduration=114.281332663 podStartE2EDuration="1m56.829917976s" totalImagesPullingTime="2.548585313s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 15:01:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 15:01:37.10240328 +0000 UTC m=+538.610277055" watchObservedRunningTime="2026-07-23 15:03:29.829917976 +0000 UTC m=+651.337791711"
Jul 23 15:03:29.831316 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:03:29.831299    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-w4bvt"]
Jul 23 15:03:29.834517 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:03:29.834503    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-w4bvt"
Jul 23 15:03:29.837679 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:03:29.837657    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Jul 23 15:03:29.837679 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:03:29.837673    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 23 15:03:29.838911 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:03:29.838895    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 23 15:03:29.838960 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:03:29.838912    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-jljcz\""
Jul 23 15:03:29.842842 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:03:29.842820    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-w4bvt"]
Jul 23 15:03:29.951958 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:03:29.951934    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88tkh\" (UniqueName: \"kubernetes.io/projected/fd4a0407-3631-43d7-8632-d5382863958f-kube-api-access-88tkh\") pod \"dns-operator-controller-manager-864dd78c4c-w4bvt\" (UID: \"fd4a0407-3631-43d7-8632-d5382863958f\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-w4bvt"
Jul 23 15:03:30.052655 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:03:30.052628    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88tkh\" (UniqueName: \"kubernetes.io/projected/fd4a0407-3631-43d7-8632-d5382863958f-kube-api-access-88tkh\") pod \"dns-operator-controller-manager-864dd78c4c-w4bvt\" (UID: \"fd4a0407-3631-43d7-8632-d5382863958f\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-w4bvt"
Jul 23 15:03:30.061053 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:03:30.061033    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-88tkh\" (UniqueName: \"kubernetes.io/projected/fd4a0407-3631-43d7-8632-d5382863958f-kube-api-access-88tkh\") pod \"dns-operator-controller-manager-864dd78c4c-w4bvt\" (UID: \"fd4a0407-3631-43d7-8632-d5382863958f\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-w4bvt"
Jul 23 15:03:30.145021 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:03:30.144959    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-w4bvt"
Jul 23 15:03:30.276324 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:03:30.276285    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-w4bvt"]
Jul 23 15:03:30.279257 ip-10-0-141-105 kubenswrapper[2560]: W0723 15:03:30.279211    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd4a0407_3631_43d7_8632_d5382863958f.slice/crio-56040707fefe070a841888ed05d51ad6fd6e2b0de4ff69a5e07227a5bfd1b6ce WatchSource:0}: Error finding container 56040707fefe070a841888ed05d51ad6fd6e2b0de4ff69a5e07227a5bfd1b6ce: Status 404 returned error can't find the container with id 56040707fefe070a841888ed05d51ad6fd6e2b0de4ff69a5e07227a5bfd1b6ce
Jul 23 15:03:30.281399 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:03:30.281384    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 23 15:03:30.549168 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:03:30.549138    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-w4bvt" event={"ID":"fd4a0407-3631-43d7-8632-d5382863958f","Type":"ContainerStarted","Data":"56040707fefe070a841888ed05d51ad6fd6e2b0de4ff69a5e07227a5bfd1b6ce"}
Jul 23 15:03:33.564057 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:03:33.564027    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-w4bvt" event={"ID":"fd4a0407-3631-43d7-8632-d5382863958f","Type":"ContainerStarted","Data":"4f27c85aa9196d566aab4e7e8ae31f41d926091820102402daaa21f71daf4882"}
Jul 23 15:03:33.564465 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:03:33.564099    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-w4bvt"
Jul 23 15:03:33.582423 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:03:33.582382    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-w4bvt" podStartSLOduration=1.662292383 podStartE2EDuration="4.582368257s" totalImagesPullingTime="2.920075874s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 15:03:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 15:03:33.581651666 +0000 UTC m=+655.089525414" watchObservedRunningTime="2026-07-23 15:03:33.582368257 +0000 UTC m=+655.090242003"
Jul 23 15:03:35.206548 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:03:35.206517    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-g6n6s"]
Jul 23 15:03:35.209719 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:03:35.209703    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-g6n6s"
Jul 23 15:03:35.212601 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:03:35.212574    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Jul 23 15:03:35.212717 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:03:35.212663    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-f9sh9\""
Jul 23 15:03:35.213362 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:03:35.213348    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Jul 23 15:03:35.219807 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:03:35.219785    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-g6n6s"]
Jul 23 15:03:35.297100 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:03:35.297073    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/c125c2da-4370-4e38-9a16-c147b743a797-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-g6n6s\" (UID: \"c125c2da-4370-4e38-9a16-c147b743a797\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-g6n6s"
Jul 23 15:03:35.297218 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:03:35.297110    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c125c2da-4370-4e38-9a16-c147b743a797-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-g6n6s\" (UID: \"c125c2da-4370-4e38-9a16-c147b743a797\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-g6n6s"
Jul 23 15:03:35.297218 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:03:35.297135    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2954\" (UniqueName: \"kubernetes.io/projected/c125c2da-4370-4e38-9a16-c147b743a797-kube-api-access-d2954\") pod \"kuadrant-console-plugin-66b5774985-g6n6s\" (UID: \"c125c2da-4370-4e38-9a16-c147b743a797\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-g6n6s"
Jul 23 15:03:35.398499 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:03:35.398471    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/c125c2da-4370-4e38-9a16-c147b743a797-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-g6n6s\" (UID: \"c125c2da-4370-4e38-9a16-c147b743a797\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-g6n6s"
Jul 23 15:03:35.398645 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:03:35.398509    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c125c2da-4370-4e38-9a16-c147b743a797-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-g6n6s\" (UID: \"c125c2da-4370-4e38-9a16-c147b743a797\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-g6n6s"
Jul 23 15:03:35.398645 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:03:35.398527    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2954\" (UniqueName: \"kubernetes.io/projected/c125c2da-4370-4e38-9a16-c147b743a797-kube-api-access-d2954\") pod \"kuadrant-console-plugin-66b5774985-g6n6s\" (UID: \"c125c2da-4370-4e38-9a16-c147b743a797\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-g6n6s"
Jul 23 15:03:35.398645 ip-10-0-141-105 kubenswrapper[2560]: E0723 15:03:35.398637    2560 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
Jul 23 15:03:35.398804 ip-10-0-141-105 kubenswrapper[2560]: E0723 15:03:35.398712    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c125c2da-4370-4e38-9a16-c147b743a797-plugin-serving-cert podName:c125c2da-4370-4e38-9a16-c147b743a797 nodeName:}" failed. No retries permitted until 2026-07-23 15:03:35.898691378 +0000 UTC m=+657.406565110 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/c125c2da-4370-4e38-9a16-c147b743a797-plugin-serving-cert") pod "kuadrant-console-plugin-66b5774985-g6n6s" (UID: "c125c2da-4370-4e38-9a16-c147b743a797") : secret "plugin-serving-cert" not found
Jul 23 15:03:35.399061 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:03:35.399042    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c125c2da-4370-4e38-9a16-c147b743a797-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-g6n6s\" (UID: \"c125c2da-4370-4e38-9a16-c147b743a797\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-g6n6s"
Jul 23 15:03:35.408408 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:03:35.408385    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2954\" (UniqueName: \"kubernetes.io/projected/c125c2da-4370-4e38-9a16-c147b743a797-kube-api-access-d2954\") pod \"kuadrant-console-plugin-66b5774985-g6n6s\" (UID: \"c125c2da-4370-4e38-9a16-c147b743a797\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-g6n6s"
Jul 23 15:03:35.904401 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:03:35.904361    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/c125c2da-4370-4e38-9a16-c147b743a797-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-g6n6s\" (UID: \"c125c2da-4370-4e38-9a16-c147b743a797\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-g6n6s"
Jul 23 15:03:35.904582 ip-10-0-141-105 kubenswrapper[2560]: E0723 15:03:35.904476    2560 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
Jul 23 15:03:35.904582 ip-10-0-141-105 kubenswrapper[2560]: E0723 15:03:35.904539    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c125c2da-4370-4e38-9a16-c147b743a797-plugin-serving-cert podName:c125c2da-4370-4e38-9a16-c147b743a797 nodeName:}" failed. No retries permitted until 2026-07-23 15:03:36.904525564 +0000 UTC m=+658.412399290 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/c125c2da-4370-4e38-9a16-c147b743a797-plugin-serving-cert") pod "kuadrant-console-plugin-66b5774985-g6n6s" (UID: "c125c2da-4370-4e38-9a16-c147b743a797") : secret "plugin-serving-cert" not found
Jul 23 15:03:36.913431 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:03:36.913398    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/c125c2da-4370-4e38-9a16-c147b743a797-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-g6n6s\" (UID: \"c125c2da-4370-4e38-9a16-c147b743a797\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-g6n6s"
Jul 23 15:03:36.915922 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:03:36.915891    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/c125c2da-4370-4e38-9a16-c147b743a797-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-g6n6s\" (UID: \"c125c2da-4370-4e38-9a16-c147b743a797\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-g6n6s"
Jul 23 15:03:37.018639 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:03:37.018604    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-g6n6s"
Jul 23 15:03:37.147699 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:03:37.147667    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-g6n6s"]
Jul 23 15:03:37.150669 ip-10-0-141-105 kubenswrapper[2560]: W0723 15:03:37.150643    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc125c2da_4370_4e38_9a16_c147b743a797.slice/crio-975ccbf46dfad16eaa0bcf4ee875d6a689726d055a640eaa6a1357a045fef8ff WatchSource:0}: Error finding container 975ccbf46dfad16eaa0bcf4ee875d6a689726d055a640eaa6a1357a045fef8ff: Status 404 returned error can't find the container with id 975ccbf46dfad16eaa0bcf4ee875d6a689726d055a640eaa6a1357a045fef8ff
Jul 23 15:03:37.582113 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:03:37.582085    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-g6n6s" event={"ID":"c125c2da-4370-4e38-9a16-c147b743a797","Type":"ContainerStarted","Data":"975ccbf46dfad16eaa0bcf4ee875d6a689726d055a640eaa6a1357a045fef8ff"}
Jul 23 15:03:44.571193 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:03:44.571152    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-w4bvt"
Jul 23 15:04:01.710293 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:04:01.710255    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-g6n6s" event={"ID":"c125c2da-4370-4e38-9a16-c147b743a797","Type":"ContainerStarted","Data":"f985a62496d820d89e03ac29fbe06924ad4e962536854bcc1f0b9f247972fb80"}
Jul 23 15:04:01.730996 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:04:01.730947    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-g6n6s" podStartSLOduration=2.835589246 podStartE2EDuration="26.730934158s" totalImagesPullingTime="23.895344912s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 15:03:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 15:04:01.728526444 +0000 UTC m=+683.236400195" watchObservedRunningTime="2026-07-23 15:04:01.730934158 +0000 UTC m=+683.238807905"
Jul 23 15:04:30.875670 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:04:30.875587    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cgwsg"]
Jul 23 15:04:30.924110 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:04:30.924068    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cgwsg"]
Jul 23 15:04:30.924110 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:04:30.924112    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cgwsg"]
Jul 23 15:04:30.924409 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:04:30.924208    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-cgwsg"
Jul 23 15:04:30.927033 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:04:30.927013    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 23 15:04:31.001157 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:04:31.001113    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pj7wz\" (UniqueName: \"kubernetes.io/projected/73e0120d-710b-45eb-9a73-83729e5a4c4a-kube-api-access-pj7wz\") pod \"limitador-limitador-54d6d988d5-cgwsg\" (UID: \"73e0120d-710b-45eb-9a73-83729e5a4c4a\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-cgwsg"
Jul 23 15:04:31.001157 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:04:31.001160    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/73e0120d-710b-45eb-9a73-83729e5a4c4a-config-file\") pod \"limitador-limitador-54d6d988d5-cgwsg\" (UID: \"73e0120d-710b-45eb-9a73-83729e5a4c4a\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-cgwsg"
Jul 23 15:04:31.102333 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:04:31.102294    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pj7wz\" (UniqueName: \"kubernetes.io/projected/73e0120d-710b-45eb-9a73-83729e5a4c4a-kube-api-access-pj7wz\") pod \"limitador-limitador-54d6d988d5-cgwsg\" (UID: \"73e0120d-710b-45eb-9a73-83729e5a4c4a\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-cgwsg"
Jul 23 15:04:31.102333 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:04:31.102337    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/73e0120d-710b-45eb-9a73-83729e5a4c4a-config-file\") pod \"limitador-limitador-54d6d988d5-cgwsg\" (UID: \"73e0120d-710b-45eb-9a73-83729e5a4c4a\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-cgwsg"
Jul 23 15:04:31.103043 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:04:31.103023    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/73e0120d-710b-45eb-9a73-83729e5a4c4a-config-file\") pod \"limitador-limitador-54d6d988d5-cgwsg\" (UID: \"73e0120d-710b-45eb-9a73-83729e5a4c4a\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-cgwsg"
Jul 23 15:04:31.111219 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:04:31.111195    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pj7wz\" (UniqueName: \"kubernetes.io/projected/73e0120d-710b-45eb-9a73-83729e5a4c4a-kube-api-access-pj7wz\") pod \"limitador-limitador-54d6d988d5-cgwsg\" (UID: \"73e0120d-710b-45eb-9a73-83729e5a4c4a\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-cgwsg"
Jul 23 15:04:31.234518 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:04:31.234483    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-cgwsg"
Jul 23 15:04:31.373845 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:04:31.373816    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cgwsg"]
Jul 23 15:04:31.377037 ip-10-0-141-105 kubenswrapper[2560]: W0723 15:04:31.377009    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73e0120d_710b_45eb_9a73_83729e5a4c4a.slice/crio-6f1c057199ac4a6359487101b809f6ea76d920930fb36980c23aa7198d5cbf99 WatchSource:0}: Error finding container 6f1c057199ac4a6359487101b809f6ea76d920930fb36980c23aa7198d5cbf99: Status 404 returned error can't find the container with id 6f1c057199ac4a6359487101b809f6ea76d920930fb36980c23aa7198d5cbf99
Jul 23 15:04:31.849320 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:04:31.849285    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-cgwsg" event={"ID":"73e0120d-710b-45eb-9a73-83729e5a4c4a","Type":"ContainerStarted","Data":"6f1c057199ac4a6359487101b809f6ea76d920930fb36980c23aa7198d5cbf99"}
Jul 23 15:04:34.864765 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:04:34.864732    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-cgwsg" event={"ID":"73e0120d-710b-45eb-9a73-83729e5a4c4a","Type":"ContainerStarted","Data":"3ae191b8c229b794ba2553350c54fe2acd17addd43a18a73f139934f63347999"}
Jul 23 15:04:34.865179 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:04:34.864836    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-cgwsg"
Jul 23 15:04:34.884840 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:04:34.884790    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-cgwsg" podStartSLOduration=2.179542215 podStartE2EDuration="4.884775141s" totalImagesPullingTime="2.705232926s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 15:04:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 15:04:34.883417189 +0000 UTC m=+716.391290937" watchObservedRunningTime="2026-07-23 15:04:34.884775141 +0000 UTC m=+716.392648912"
Jul 23 15:04:45.871162 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:04:45.871134    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-cgwsg"
Jul 23 15:05:02.321383 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:05:02.321355    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-lng5p"]
Jul 23 15:05:02.323647 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:05:02.323631    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-lng5p"
Jul 23 15:05:02.326438 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:05:02.326418    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-cn46q\""
Jul 23 15:05:02.332198 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:05:02.332180    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-lng5p"]
Jul 23 15:05:02.371082 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:05:02.371061    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdsvq\" (UniqueName: \"kubernetes.io/projected/fc7e850f-5e33-4b7e-afbe-61316dc72fdd-kube-api-access-fdsvq\") pod \"authorino-5d4b67c446-lng5p\" (UID: \"fc7e850f-5e33-4b7e-afbe-61316dc72fdd\") " pod="kuadrant-system/authorino-5d4b67c446-lng5p"
Jul 23 15:05:02.471578 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:05:02.471555    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdsvq\" (UniqueName: \"kubernetes.io/projected/fc7e850f-5e33-4b7e-afbe-61316dc72fdd-kube-api-access-fdsvq\") pod \"authorino-5d4b67c446-lng5p\" (UID: \"fc7e850f-5e33-4b7e-afbe-61316dc72fdd\") " pod="kuadrant-system/authorino-5d4b67c446-lng5p"
Jul 23 15:05:02.481910 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:05:02.481884    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdsvq\" (UniqueName: \"kubernetes.io/projected/fc7e850f-5e33-4b7e-afbe-61316dc72fdd-kube-api-access-fdsvq\") pod \"authorino-5d4b67c446-lng5p\" (UID: \"fc7e850f-5e33-4b7e-afbe-61316dc72fdd\") " pod="kuadrant-system/authorino-5d4b67c446-lng5p"
Jul 23 15:05:02.580691 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:05:02.580634    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-lng5p"]
Jul 23 15:05:02.580851 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:05:02.580837    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-lng5p"
Jul 23 15:05:02.707609 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:05:02.707588    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-lng5p"]
Jul 23 15:05:02.709556 ip-10-0-141-105 kubenswrapper[2560]: W0723 15:05:02.709529    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfc7e850f_5e33_4b7e_afbe_61316dc72fdd.slice/crio-6496dfee2b4e2721b01359d85e0f4458562bd6598503c5bada14cd33765049f3 WatchSource:0}: Error finding container 6496dfee2b4e2721b01359d85e0f4458562bd6598503c5bada14cd33765049f3: Status 404 returned error can't find the container with id 6496dfee2b4e2721b01359d85e0f4458562bd6598503c5bada14cd33765049f3
Jul 23 15:05:02.985755 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:05:02.985724    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-lng5p" event={"ID":"fc7e850f-5e33-4b7e-afbe-61316dc72fdd","Type":"ContainerStarted","Data":"6496dfee2b4e2721b01359d85e0f4458562bd6598503c5bada14cd33765049f3"}
Jul 23 15:05:07.005523 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:05:07.005488    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-lng5p" event={"ID":"fc7e850f-5e33-4b7e-afbe-61316dc72fdd","Type":"ContainerStarted","Data":"edceb4a11ca564450dd8608ab8af90d21689181a02ff658c1438c3edd8054749"}
Jul 23 15:05:07.005980 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:05:07.005541    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5d4b67c446-lng5p" podUID="fc7e850f-5e33-4b7e-afbe-61316dc72fdd" containerName="authorino" containerID="cri-o://edceb4a11ca564450dd8608ab8af90d21689181a02ff658c1438c3edd8054749" gracePeriod=30
Jul 23 15:05:07.022673 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:05:07.022630    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5d4b67c446-lng5p" podStartSLOduration=1.338986873 podStartE2EDuration="5.022616794s" totalImagesPullingTime="3.683629921s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 15:05:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 15:05:07.021076772 +0000 UTC m=+748.528950522" watchObservedRunningTime="2026-07-23 15:05:07.022616794 +0000 UTC m=+748.530490542"
Jul 23 15:05:07.253216 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:05:07.253195    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-lng5p"
Jul 23 15:05:07.314292 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:05:07.314194    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fdsvq\" (UniqueName: \"kubernetes.io/projected/fc7e850f-5e33-4b7e-afbe-61316dc72fdd-kube-api-access-fdsvq\") pod \"fc7e850f-5e33-4b7e-afbe-61316dc72fdd\" (UID: \"fc7e850f-5e33-4b7e-afbe-61316dc72fdd\") "
Jul 23 15:05:07.316322 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:05:07.316299    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc7e850f-5e33-4b7e-afbe-61316dc72fdd-kube-api-access-fdsvq" (OuterVolumeSpecName: "kube-api-access-fdsvq") pod "fc7e850f-5e33-4b7e-afbe-61316dc72fdd" (UID: "fc7e850f-5e33-4b7e-afbe-61316dc72fdd"). InnerVolumeSpecName "kube-api-access-fdsvq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 15:05:07.415478 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:05:07.415455    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdsvq\" (UniqueName: \"kubernetes.io/projected/fc7e850f-5e33-4b7e-afbe-61316dc72fdd-kube-api-access-fdsvq\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Jul 23 15:05:08.011425 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:05:08.011391    2560 generic.go:356] "Generic (PLEG): container finished" podID="fc7e850f-5e33-4b7e-afbe-61316dc72fdd" containerID="edceb4a11ca564450dd8608ab8af90d21689181a02ff658c1438c3edd8054749" exitCode=0
Jul 23 15:05:08.012005 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:05:08.011440    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-lng5p"
Jul 23 15:05:08.012005 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:05:08.011484    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-lng5p" event={"ID":"fc7e850f-5e33-4b7e-afbe-61316dc72fdd","Type":"ContainerDied","Data":"edceb4a11ca564450dd8608ab8af90d21689181a02ff658c1438c3edd8054749"}
Jul 23 15:05:08.012005 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:05:08.011529    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-lng5p" event={"ID":"fc7e850f-5e33-4b7e-afbe-61316dc72fdd","Type":"ContainerDied","Data":"6496dfee2b4e2721b01359d85e0f4458562bd6598503c5bada14cd33765049f3"}
Jul 23 15:05:08.012005 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:05:08.011553    2560 scope.go:117] "RemoveContainer" containerID="edceb4a11ca564450dd8608ab8af90d21689181a02ff658c1438c3edd8054749"
Jul 23 15:05:08.020382 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:05:08.020367    2560 scope.go:117] "RemoveContainer" containerID="edceb4a11ca564450dd8608ab8af90d21689181a02ff658c1438c3edd8054749"
Jul 23 15:05:08.020615 ip-10-0-141-105 kubenswrapper[2560]: E0723 15:05:08.020596    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edceb4a11ca564450dd8608ab8af90d21689181a02ff658c1438c3edd8054749\": container with ID starting with edceb4a11ca564450dd8608ab8af90d21689181a02ff658c1438c3edd8054749 not found: ID does not exist" containerID="edceb4a11ca564450dd8608ab8af90d21689181a02ff658c1438c3edd8054749"
Jul 23 15:05:08.020665 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:05:08.020622    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edceb4a11ca564450dd8608ab8af90d21689181a02ff658c1438c3edd8054749"} err="failed to get container status \"edceb4a11ca564450dd8608ab8af90d21689181a02ff658c1438c3edd8054749\": rpc error: code = NotFound desc = could not find container \"edceb4a11ca564450dd8608ab8af90d21689181a02ff658c1438c3edd8054749\": container with ID starting with edceb4a11ca564450dd8608ab8af90d21689181a02ff658c1438c3edd8054749 not found: ID does not exist"
Jul 23 15:05:08.033109 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:05:08.033080    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-lng5p"]
Jul 23 15:05:08.039108 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:05:08.039088    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-lng5p"]
Jul 23 15:05:09.061695 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:05:09.061662    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc7e850f-5e33-4b7e-afbe-61316dc72fdd" path="/var/lib/kubelet/pods/fc7e850f-5e33-4b7e-afbe-61316dc72fdd/volumes"
Jul 23 15:05:46.831529 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:05:46.831492    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cgwsg"]
Jul 23 15:06:05.168442 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:06:05.168367    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cgwsg"]
Jul 23 15:06:19.541968 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:06:19.541882    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cgwsg"]
Jul 23 15:06:28.331490 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:06:28.331459    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-9htj9"]
Jul 23 15:06:28.331880 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:06:28.331787    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc7e850f-5e33-4b7e-afbe-61316dc72fdd" containerName="authorino"
Jul 23 15:06:28.331880 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:06:28.331798    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc7e850f-5e33-4b7e-afbe-61316dc72fdd" containerName="authorino"
Jul 23 15:06:28.331880 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:06:28.331875    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc7e850f-5e33-4b7e-afbe-61316dc72fdd" containerName="authorino"
Jul 23 15:06:28.334025 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:06:28.334009    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9htj9"
Jul 23 15:06:28.338280 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:06:28.338209    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 23 15:06:28.338404 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:06:28.338329    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 23 15:06:28.338404 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:06:28.338378    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-nnqmb\""
Jul 23 15:06:28.338523 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:06:28.338425    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Jul 23 15:06:28.347035 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:06:28.347007    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-9htj9"]
Jul 23 15:06:28.430011 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:06:28.429973    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bm6hl\" (UniqueName: \"kubernetes.io/projected/36060547-3588-4f78-b737-714d88f73dc3-kube-api-access-bm6hl\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9htj9\" (UID: \"36060547-3588-4f78-b737-714d88f73dc3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9htj9"
Jul 23 15:06:28.430011 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:06:28.430017    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36060547-3588-4f78-b737-714d88f73dc3-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9htj9\" (UID: \"36060547-3588-4f78-b737-714d88f73dc3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9htj9"
Jul 23 15:06:28.430221 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:06:28.430039    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36060547-3588-4f78-b737-714d88f73dc3-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9htj9\" (UID: \"36060547-3588-4f78-b737-714d88f73dc3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9htj9"
Jul 23 15:06:28.430221 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:06:28.430129    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36060547-3588-4f78-b737-714d88f73dc3-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9htj9\" (UID: \"36060547-3588-4f78-b737-714d88f73dc3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9htj9"
Jul 23 15:06:28.430221 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:06:28.430180    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36060547-3588-4f78-b737-714d88f73dc3-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9htj9\" (UID: \"36060547-3588-4f78-b737-714d88f73dc3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9htj9"
Jul 23 15:06:28.430221 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:06:28.430207    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36060547-3588-4f78-b737-714d88f73dc3-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9htj9\" (UID: \"36060547-3588-4f78-b737-714d88f73dc3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9htj9"
Jul 23 15:06:28.531322 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:06:28.531280    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bm6hl\" (UniqueName: \"kubernetes.io/projected/36060547-3588-4f78-b737-714d88f73dc3-kube-api-access-bm6hl\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9htj9\" (UID: \"36060547-3588-4f78-b737-714d88f73dc3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9htj9"
Jul 23 15:06:28.531460 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:06:28.531331    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36060547-3588-4f78-b737-714d88f73dc3-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9htj9\" (UID: \"36060547-3588-4f78-b737-714d88f73dc3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9htj9"
Jul 23 15:06:28.531501 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:06:28.531464    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36060547-3588-4f78-b737-714d88f73dc3-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9htj9\" (UID: \"36060547-3588-4f78-b737-714d88f73dc3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9htj9"
Jul 23 15:06:28.531555 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:06:28.531522    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36060547-3588-4f78-b737-714d88f73dc3-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9htj9\" (UID: \"36060547-3588-4f78-b737-714d88f73dc3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9htj9"
Jul 23 15:06:28.531594 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:06:28.531569    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36060547-3588-4f78-b737-714d88f73dc3-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9htj9\" (UID: \"36060547-3588-4f78-b737-714d88f73dc3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9htj9"
Jul 23 15:06:28.531644 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:06:28.531591    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36060547-3588-4f78-b737-714d88f73dc3-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9htj9\" (UID: \"36060547-3588-4f78-b737-714d88f73dc3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9htj9"
Jul 23 15:06:28.531742 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:06:28.531723    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36060547-3588-4f78-b737-714d88f73dc3-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9htj9\" (UID: \"36060547-3588-4f78-b737-714d88f73dc3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9htj9"
Jul 23 15:06:28.531810 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:06:28.531763    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36060547-3588-4f78-b737-714d88f73dc3-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9htj9\" (UID: \"36060547-3588-4f78-b737-714d88f73dc3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9htj9"
Jul 23 15:06:28.531906 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:06:28.531887    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36060547-3588-4f78-b737-714d88f73dc3-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9htj9\" (UID: \"36060547-3588-4f78-b737-714d88f73dc3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9htj9"
Jul 23 15:06:28.533980 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:06:28.533962    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36060547-3588-4f78-b737-714d88f73dc3-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9htj9\" (UID: \"36060547-3588-4f78-b737-714d88f73dc3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9htj9"
Jul 23 15:06:28.534363 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:06:28.534343    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36060547-3588-4f78-b737-714d88f73dc3-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9htj9\" (UID: \"36060547-3588-4f78-b737-714d88f73dc3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9htj9"
Jul 23 15:06:28.541077 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:06:28.541054    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bm6hl\" (UniqueName: \"kubernetes.io/projected/36060547-3588-4f78-b737-714d88f73dc3-kube-api-access-bm6hl\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9htj9\" (UID: \"36060547-3588-4f78-b737-714d88f73dc3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9htj9"
Jul 23 15:06:28.644075 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:06:28.643985    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9htj9"
Jul 23 15:06:28.776486 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:06:28.776424    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-9htj9"]
Jul 23 15:06:28.779324 ip-10-0-141-105 kubenswrapper[2560]: W0723 15:06:28.779281    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36060547_3588_4f78_b737_714d88f73dc3.slice/crio-08b0bf8ecb5c1aa40c1a86be25b6881cb7fd9e0bb2ff74b7aa70e57e01a82832 WatchSource:0}: Error finding container 08b0bf8ecb5c1aa40c1a86be25b6881cb7fd9e0bb2ff74b7aa70e57e01a82832: Status 404 returned error can't find the container with id 08b0bf8ecb5c1aa40c1a86be25b6881cb7fd9e0bb2ff74b7aa70e57e01a82832
Jul 23 15:06:29.369963 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:06:29.369914    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9htj9" event={"ID":"36060547-3588-4f78-b737-714d88f73dc3","Type":"ContainerStarted","Data":"08b0bf8ecb5c1aa40c1a86be25b6881cb7fd9e0bb2ff74b7aa70e57e01a82832"}
Jul 23 15:06:29.434003 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:06:29.433955    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cgwsg"]
Jul 23 15:06:30.376166 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:06:30.376137    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9htj9" event={"ID":"36060547-3588-4f78-b737-714d88f73dc3","Type":"ContainerStarted","Data":"2956db150a4ae9ec9817128ebb5e2736c156b848d3cc4aa198ac86f3cd38a6e6"}
Jul 23 15:06:30.396292 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:06:30.396247    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9htj9" podStartSLOduration=1.325454656 podStartE2EDuration="2.396216794s" totalImagesPullingTime="1.070762138s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 15:06:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 15:06:30.394483275 +0000 UTC m=+831.902357023" watchObservedRunningTime="2026-07-23 15:06:30.396216794 +0000 UTC m=+831.904090533"
Jul 23 15:06:37.630256 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:06:37.630190    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cgwsg"]
Jul 23 15:06:38.644624 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:06:38.644589    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9htj9"
Jul 23 15:06:38.645003 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:06:38.644668    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9htj9"
Jul 23 15:06:38.657112 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:06:38.657083    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9htj9"
Jul 23 15:06:39.435211 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:06:39.435178    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9htj9"
Jul 23 15:06:48.824366 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:06:48.824333    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-zvpnp"]
Jul 23 15:06:48.827692 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:06:48.827674    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zvpnp"
Jul 23 15:06:48.830565 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:06:48.830547    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Jul 23 15:06:48.836357 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:06:48.836333    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-zvpnp"]
Jul 23 15:06:48.913779 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:06:48.913745    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22c9c0ea-6dc7-4915-bc33-ab6ace48274e-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zvpnp\" (UID: \"22c9c0ea-6dc7-4915-bc33-ab6ace48274e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zvpnp"
Jul 23 15:06:48.913945 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:06:48.913806    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22c9c0ea-6dc7-4915-bc33-ab6ace48274e-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zvpnp\" (UID: \"22c9c0ea-6dc7-4915-bc33-ab6ace48274e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zvpnp"
Jul 23 15:06:48.913945 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:06:48.913859    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22c9c0ea-6dc7-4915-bc33-ab6ace48274e-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zvpnp\" (UID: \"22c9c0ea-6dc7-4915-bc33-ab6ace48274e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zvpnp"
Jul 23 15:06:48.913945 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:06:48.913888    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45nq2\" (UniqueName: \"kubernetes.io/projected/22c9c0ea-6dc7-4915-bc33-ab6ace48274e-kube-api-access-45nq2\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zvpnp\" (UID: \"22c9c0ea-6dc7-4915-bc33-ab6ace48274e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zvpnp"
Jul 23 15:06:48.913945 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:06:48.913935    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22c9c0ea-6dc7-4915-bc33-ab6ace48274e-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zvpnp\" (UID: \"22c9c0ea-6dc7-4915-bc33-ab6ace48274e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zvpnp"
Jul 23 15:06:48.914073 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:06:48.913960    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22c9c0ea-6dc7-4915-bc33-ab6ace48274e-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zvpnp\" (UID: \"22c9c0ea-6dc7-4915-bc33-ab6ace48274e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zvpnp"
Jul 23 15:06:49.015445 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:06:49.015407    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22c9c0ea-6dc7-4915-bc33-ab6ace48274e-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zvpnp\" (UID: \"22c9c0ea-6dc7-4915-bc33-ab6ace48274e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zvpnp"
Jul 23 15:06:49.015445 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:06:49.015451    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45nq2\" (UniqueName: \"kubernetes.io/projected/22c9c0ea-6dc7-4915-bc33-ab6ace48274e-kube-api-access-45nq2\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zvpnp\" (UID: \"22c9c0ea-6dc7-4915-bc33-ab6ace48274e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zvpnp"
Jul 23 15:06:49.015660 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:06:49.015494    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22c9c0ea-6dc7-4915-bc33-ab6ace48274e-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zvpnp\" (UID: \"22c9c0ea-6dc7-4915-bc33-ab6ace48274e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zvpnp"
Jul 23 15:06:49.015660 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:06:49.015512    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22c9c0ea-6dc7-4915-bc33-ab6ace48274e-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zvpnp\" (UID: \"22c9c0ea-6dc7-4915-bc33-ab6ace48274e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zvpnp"
Jul 23 15:06:49.015660 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:06:49.015540    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22c9c0ea-6dc7-4915-bc33-ab6ace48274e-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zvpnp\" (UID: \"22c9c0ea-6dc7-4915-bc33-ab6ace48274e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zvpnp"
Jul 23 15:06:49.015805 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:06:49.015707    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22c9c0ea-6dc7-4915-bc33-ab6ace48274e-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zvpnp\" (UID: \"22c9c0ea-6dc7-4915-bc33-ab6ace48274e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zvpnp"
Jul 23 15:06:49.015844 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:06:49.015827    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22c9c0ea-6dc7-4915-bc33-ab6ace48274e-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zvpnp\" (UID: \"22c9c0ea-6dc7-4915-bc33-ab6ace48274e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zvpnp"
Jul 23 15:06:49.015907 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:06:49.015886    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22c9c0ea-6dc7-4915-bc33-ab6ace48274e-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zvpnp\" (UID: \"22c9c0ea-6dc7-4915-bc33-ab6ace48274e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zvpnp"
Jul 23 15:06:49.015994 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:06:49.015977    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22c9c0ea-6dc7-4915-bc33-ab6ace48274e-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zvpnp\" (UID: \"22c9c0ea-6dc7-4915-bc33-ab6ace48274e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zvpnp"
Jul 23 15:06:49.017960 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:06:49.017942    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22c9c0ea-6dc7-4915-bc33-ab6ace48274e-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zvpnp\" (UID: \"22c9c0ea-6dc7-4915-bc33-ab6ace48274e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zvpnp"
Jul 23 15:06:49.018080 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:06:49.018063    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22c9c0ea-6dc7-4915-bc33-ab6ace48274e-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zvpnp\" (UID: \"22c9c0ea-6dc7-4915-bc33-ab6ace48274e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zvpnp"
Jul 23 15:06:49.024770 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:06:49.024744    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-45nq2\" (UniqueName: \"kubernetes.io/projected/22c9c0ea-6dc7-4915-bc33-ab6ace48274e-kube-api-access-45nq2\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zvpnp\" (UID: \"22c9c0ea-6dc7-4915-bc33-ab6ace48274e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zvpnp"
Jul 23 15:06:49.136498 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:06:49.136409    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zvpnp"
Jul 23 15:06:49.269772 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:06:49.269744    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-zvpnp"]
Jul 23 15:06:49.272199 ip-10-0-141-105 kubenswrapper[2560]: W0723 15:06:49.272164    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod22c9c0ea_6dc7_4915_bc33_ab6ace48274e.slice/crio-5ee194aee0b25e7165347d2f7acfbe2d633caa075c555bf33edc4fea0fb96a30 WatchSource:0}: Error finding container 5ee194aee0b25e7165347d2f7acfbe2d633caa075c555bf33edc4fea0fb96a30: Status 404 returned error can't find the container with id 5ee194aee0b25e7165347d2f7acfbe2d633caa075c555bf33edc4fea0fb96a30
Jul 23 15:06:49.466764 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:06:49.466733    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zvpnp" event={"ID":"22c9c0ea-6dc7-4915-bc33-ab6ace48274e","Type":"ContainerStarted","Data":"5ee194aee0b25e7165347d2f7acfbe2d633caa075c555bf33edc4fea0fb96a30"}
Jul 23 15:06:50.476186 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:06:50.476148    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zvpnp" event={"ID":"22c9c0ea-6dc7-4915-bc33-ab6ace48274e","Type":"ContainerStarted","Data":"42bbaaf1dd90ad72cca05a5a91f0d51f0ba81ca43b3492d4c4489fed47aa4c50"}
Jul 23 15:06:50.494828 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:06:50.494784    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zvpnp" podStartSLOduration=2.154238606 podStartE2EDuration="2.494767588s" totalImagesPullingTime="340.528982ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 15:06:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 15:06:50.494314176 +0000 UTC m=+852.002188015" watchObservedRunningTime="2026-07-23 15:06:50.494767588 +0000 UTC m=+852.002641336"
Jul 23 15:06:51.135292 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:06:51.135251    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cgwsg"]
Jul 23 15:06:59.136554 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:06:59.136520    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zvpnp"
Jul 23 15:06:59.136554 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:06:59.136563    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zvpnp"
Jul 23 15:06:59.151054 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:06:59.151025    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zvpnp"
Jul 23 15:06:59.527255 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:06:59.527199    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zvpnp"
Jul 23 15:07:39.982384 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:07:39.982347    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6777b9bd49-vb587"]
Jul 23 15:07:39.985602 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:07:39.985585    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6777b9bd49-vb587"
Jul 23 15:07:39.989635 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:07:39.989605    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-cn46q\""
Jul 23 15:07:39.989764 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:07:39.989618    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Jul 23 15:07:39.994125 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:07:39.994098    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6777b9bd49-vb587"]
Jul 23 15:07:40.042541 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:07:40.042502    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pc598\" (UniqueName: \"kubernetes.io/projected/36f2ed03-9cc1-421b-821a-140db7c0a1c6-kube-api-access-pc598\") pod \"authorino-6777b9bd49-vb587\" (UID: \"36f2ed03-9cc1-421b-821a-140db7c0a1c6\") " pod="kuadrant-system/authorino-6777b9bd49-vb587"
Jul 23 15:07:40.042541 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:07:40.042547    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/36f2ed03-9cc1-421b-821a-140db7c0a1c6-tls-cert\") pod \"authorino-6777b9bd49-vb587\" (UID: \"36f2ed03-9cc1-421b-821a-140db7c0a1c6\") " pod="kuadrant-system/authorino-6777b9bd49-vb587"
Jul 23 15:07:40.143163 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:07:40.143023    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pc598\" (UniqueName: \"kubernetes.io/projected/36f2ed03-9cc1-421b-821a-140db7c0a1c6-kube-api-access-pc598\") pod \"authorino-6777b9bd49-vb587\" (UID: \"36f2ed03-9cc1-421b-821a-140db7c0a1c6\") " pod="kuadrant-system/authorino-6777b9bd49-vb587"
Jul 23 15:07:40.143163 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:07:40.143067    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/36f2ed03-9cc1-421b-821a-140db7c0a1c6-tls-cert\") pod \"authorino-6777b9bd49-vb587\" (UID: \"36f2ed03-9cc1-421b-821a-140db7c0a1c6\") " pod="kuadrant-system/authorino-6777b9bd49-vb587"
Jul 23 15:07:40.149369 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:07:40.146006    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/36f2ed03-9cc1-421b-821a-140db7c0a1c6-tls-cert\") pod \"authorino-6777b9bd49-vb587\" (UID: \"36f2ed03-9cc1-421b-821a-140db7c0a1c6\") " pod="kuadrant-system/authorino-6777b9bd49-vb587"
Jul 23 15:07:40.152703 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:07:40.152681    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pc598\" (UniqueName: \"kubernetes.io/projected/36f2ed03-9cc1-421b-821a-140db7c0a1c6-kube-api-access-pc598\") pod \"authorino-6777b9bd49-vb587\" (UID: \"36f2ed03-9cc1-421b-821a-140db7c0a1c6\") " pod="kuadrant-system/authorino-6777b9bd49-vb587"
Jul 23 15:07:40.163014 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:07:40.162990    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z4fvn_e0c66f26-06fe-4273-ba29-fa20ce89b873/ovn-acl-logging/0.log"
Jul 23 15:07:40.165565 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:07:40.165543    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z4fvn_e0c66f26-06fe-4273-ba29-fa20ce89b873/ovn-acl-logging/0.log"
Jul 23 15:07:40.294971 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:07:40.294881    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6777b9bd49-vb587"
Jul 23 15:07:40.426641 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:07:40.426446    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6777b9bd49-vb587"]
Jul 23 15:07:40.704443 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:07:40.704350    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6777b9bd49-vb587" event={"ID":"36f2ed03-9cc1-421b-821a-140db7c0a1c6","Type":"ContainerStarted","Data":"a94f4aa23b730ead5b9685a4b9ee799d7978be91bb2a595c1a385778702c36e7"}
Jul 23 15:07:41.710997 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:07:41.710964    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6777b9bd49-vb587" event={"ID":"36f2ed03-9cc1-421b-821a-140db7c0a1c6","Type":"ContainerStarted","Data":"76377cbbb994edc0a87104883c960499d13c4bc765d1913be2d5807be1571447"}
Jul 23 15:07:41.729327 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:07:41.729261    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-6777b9bd49-vb587" podStartSLOduration=1.699703522 podStartE2EDuration="2.729224227s" totalImagesPullingTime="1.029520705s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 15:07:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 15:07:41.72833421 +0000 UTC m=+903.236207956" watchObservedRunningTime="2026-07-23 15:07:41.729224227 +0000 UTC m=+903.237097973"
Jul 23 15:08:15.240739 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:08:15.240700    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cgwsg"]
Jul 23 15:08:25.332255 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:08:25.332199    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cgwsg"]
Jul 23 15:08:36.192760 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:08:36.192724    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cgwsg"]
Jul 23 15:10:17.038559 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:10:17.038517    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cgwsg"]
Jul 23 15:10:29.523470 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:10:29.523380    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cgwsg"]
Jul 23 15:11:03.623978 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:11:03.623940    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cgwsg"]
Jul 23 15:11:13.519823 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:11:13.519781    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cgwsg"]
Jul 23 15:11:21.334435 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:11:21.334402    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cgwsg"]
Jul 23 15:11:30.531374 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:11:30.531338    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cgwsg"]
Jul 23 15:11:41.227819 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:11:41.227781    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cgwsg"]
Jul 23 15:11:50.331749 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:11:50.331708    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cgwsg"]
Jul 23 15:12:02.732684 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:12:02.732590    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cgwsg"]
Jul 23 15:12:12.929308 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:12:12.929269    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cgwsg"]
Jul 23 15:12:38.333589 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:12:38.333545    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cgwsg"]
Jul 23 15:12:40.198373 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:12:40.198270    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z4fvn_e0c66f26-06fe-4273-ba29-fa20ce89b873/ovn-acl-logging/0.log"
Jul 23 15:12:40.198373 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:12:40.198352    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z4fvn_e0c66f26-06fe-4273-ba29-fa20ce89b873/ovn-acl-logging/0.log"
Jul 23 15:12:50.041320 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:12:50.041285    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cgwsg"]
Jul 23 15:13:13.720784 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:13:13.720748    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cgwsg"]
Jul 23 15:13:24.936122 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:13:24.936086    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cgwsg"]
Jul 23 15:13:41.532854 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:13:41.532752    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cgwsg"]
Jul 23 15:13:50.829113 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:13:50.829077    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cgwsg"]
Jul 23 15:14:00.435275 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:14:00.435241    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cgwsg"]
Jul 23 15:14:10.326634 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:14:10.326597    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cgwsg"]
Jul 23 15:14:19.332244 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:14:19.332203    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cgwsg"]
Jul 23 15:16:14.126844 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:16:14.126807    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cgwsg"]
Jul 23 15:16:23.424086 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:16:23.424048    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cgwsg"]
Jul 23 15:16:36.333932 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:16:36.333854    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cgwsg"]
Jul 23 15:16:47.731026 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:16:47.730986    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cgwsg"]
Jul 23 15:16:58.420855 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:16:58.420815    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cgwsg"]
Jul 23 15:17:09.526337 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:17:09.526303    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cgwsg"]
Jul 23 15:17:19.229752 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:17:19.229718    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cgwsg"]
Jul 23 15:17:28.822791 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:17:28.822754    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cgwsg"]
Jul 23 15:17:40.227565 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:17:40.227527    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z4fvn_e0c66f26-06fe-4273-ba29-fa20ce89b873/ovn-acl-logging/0.log"
Jul 23 15:17:40.229843 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:17:40.228616    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z4fvn_e0c66f26-06fe-4273-ba29-fa20ce89b873/ovn-acl-logging/0.log"
Jul 23 15:17:40.529099 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:17:40.528994    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cgwsg"]
Jul 23 15:17:48.239116 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:17:48.239061    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cgwsg"]
Jul 23 15:17:57.431772 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:17:57.431741    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cgwsg"]
Jul 23 15:18:06.334141 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:18:06.334051    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cgwsg"]
Jul 23 15:18:24.337816 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:18:24.337772    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cgwsg"]
Jul 23 15:18:34.429411 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:18:34.429374    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cgwsg"]
Jul 23 15:18:35.741997 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:18:35.741965    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-9e116000-kserve-7969f5cbf5-ltt6l"]
Jul 23 15:18:35.745357 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:18:35.745340    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-9e116000-kserve-7969f5cbf5-ltt6l"
Jul 23 15:18:35.748203 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:18:35.748185    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-9e116000-kserve-self-signed-certs\""
Jul 23 15:18:35.761382 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:18:35.761357    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-9e116000-kserve-7969f5cbf5-ltt6l"]
Jul 23 15:18:35.809804 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:18:35.809769    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40ef677a-3e8c-41aa-8f8e-a2661ed670ec-model-cache\") pod \"e2e-same-id-a-9e116000-kserve-7969f5cbf5-ltt6l\" (UID: \"40ef677a-3e8c-41aa-8f8e-a2661ed670ec\") " pod="llm/e2e-same-id-a-9e116000-kserve-7969f5cbf5-ltt6l"
Jul 23 15:18:35.809997 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:18:35.809822    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40ef677a-3e8c-41aa-8f8e-a2661ed670ec-home\") pod \"e2e-same-id-a-9e116000-kserve-7969f5cbf5-ltt6l\" (UID: \"40ef677a-3e8c-41aa-8f8e-a2661ed670ec\") " pod="llm/e2e-same-id-a-9e116000-kserve-7969f5cbf5-ltt6l"
Jul 23 15:18:35.809997 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:18:35.809845    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztd9z\" (UniqueName: \"kubernetes.io/projected/40ef677a-3e8c-41aa-8f8e-a2661ed670ec-kube-api-access-ztd9z\") pod \"e2e-same-id-a-9e116000-kserve-7969f5cbf5-ltt6l\" (UID: \"40ef677a-3e8c-41aa-8f8e-a2661ed670ec\") " pod="llm/e2e-same-id-a-9e116000-kserve-7969f5cbf5-ltt6l"
Jul 23 15:18:35.809997 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:18:35.809911    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40ef677a-3e8c-41aa-8f8e-a2661ed670ec-tmp-dir\") pod \"e2e-same-id-a-9e116000-kserve-7969f5cbf5-ltt6l\" (UID: \"40ef677a-3e8c-41aa-8f8e-a2661ed670ec\") " pod="llm/e2e-same-id-a-9e116000-kserve-7969f5cbf5-ltt6l"
Jul 23 15:18:35.809997 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:18:35.809940    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40ef677a-3e8c-41aa-8f8e-a2661ed670ec-tls-certs\") pod \"e2e-same-id-a-9e116000-kserve-7969f5cbf5-ltt6l\" (UID: \"40ef677a-3e8c-41aa-8f8e-a2661ed670ec\") " pod="llm/e2e-same-id-a-9e116000-kserve-7969f5cbf5-ltt6l"
Jul 23 15:18:35.809997 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:18:35.809964    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40ef677a-3e8c-41aa-8f8e-a2661ed670ec-dshm\") pod \"e2e-same-id-a-9e116000-kserve-7969f5cbf5-ltt6l\" (UID: \"40ef677a-3e8c-41aa-8f8e-a2661ed670ec\") " pod="llm/e2e-same-id-a-9e116000-kserve-7969f5cbf5-ltt6l"
Jul 23 15:18:35.910956 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:18:35.910912    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40ef677a-3e8c-41aa-8f8e-a2661ed670ec-home\") pod \"e2e-same-id-a-9e116000-kserve-7969f5cbf5-ltt6l\" (UID: \"40ef677a-3e8c-41aa-8f8e-a2661ed670ec\") " pod="llm/e2e-same-id-a-9e116000-kserve-7969f5cbf5-ltt6l"
Jul 23 15:18:35.910956 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:18:35.910949    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztd9z\" (UniqueName: \"kubernetes.io/projected/40ef677a-3e8c-41aa-8f8e-a2661ed670ec-kube-api-access-ztd9z\") pod \"e2e-same-id-a-9e116000-kserve-7969f5cbf5-ltt6l\" (UID: \"40ef677a-3e8c-41aa-8f8e-a2661ed670ec\") " pod="llm/e2e-same-id-a-9e116000-kserve-7969f5cbf5-ltt6l"
Jul 23 15:18:35.911271 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:18:35.911088    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40ef677a-3e8c-41aa-8f8e-a2661ed670ec-tmp-dir\") pod \"e2e-same-id-a-9e116000-kserve-7969f5cbf5-ltt6l\" (UID: \"40ef677a-3e8c-41aa-8f8e-a2661ed670ec\") " pod="llm/e2e-same-id-a-9e116000-kserve-7969f5cbf5-ltt6l"
Jul 23 15:18:35.911271 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:18:35.911121    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40ef677a-3e8c-41aa-8f8e-a2661ed670ec-tls-certs\") pod \"e2e-same-id-a-9e116000-kserve-7969f5cbf5-ltt6l\" (UID: \"40ef677a-3e8c-41aa-8f8e-a2661ed670ec\") " pod="llm/e2e-same-id-a-9e116000-kserve-7969f5cbf5-ltt6l"
Jul 23 15:18:35.911271 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:18:35.911160    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40ef677a-3e8c-41aa-8f8e-a2661ed670ec-dshm\") pod \"e2e-same-id-a-9e116000-kserve-7969f5cbf5-ltt6l\" (UID: \"40ef677a-3e8c-41aa-8f8e-a2661ed670ec\") " pod="llm/e2e-same-id-a-9e116000-kserve-7969f5cbf5-ltt6l"
Jul 23 15:18:35.911458 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:18:35.911293    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40ef677a-3e8c-41aa-8f8e-a2661ed670ec-model-cache\") pod \"e2e-same-id-a-9e116000-kserve-7969f5cbf5-ltt6l\" (UID: \"40ef677a-3e8c-41aa-8f8e-a2661ed670ec\") " pod="llm/e2e-same-id-a-9e116000-kserve-7969f5cbf5-ltt6l"
Jul 23 15:18:35.911458 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:18:35.911357    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40ef677a-3e8c-41aa-8f8e-a2661ed670ec-home\") pod \"e2e-same-id-a-9e116000-kserve-7969f5cbf5-ltt6l\" (UID: \"40ef677a-3e8c-41aa-8f8e-a2661ed670ec\") " pod="llm/e2e-same-id-a-9e116000-kserve-7969f5cbf5-ltt6l"
Jul 23 15:18:35.911562 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:18:35.911473    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40ef677a-3e8c-41aa-8f8e-a2661ed670ec-tmp-dir\") pod \"e2e-same-id-a-9e116000-kserve-7969f5cbf5-ltt6l\" (UID: \"40ef677a-3e8c-41aa-8f8e-a2661ed670ec\") " pod="llm/e2e-same-id-a-9e116000-kserve-7969f5cbf5-ltt6l"
Jul 23 15:18:35.911648 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:18:35.911620    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40ef677a-3e8c-41aa-8f8e-a2661ed670ec-model-cache\") pod \"e2e-same-id-a-9e116000-kserve-7969f5cbf5-ltt6l\" (UID: \"40ef677a-3e8c-41aa-8f8e-a2661ed670ec\") " pod="llm/e2e-same-id-a-9e116000-kserve-7969f5cbf5-ltt6l"
Jul 23 15:18:35.913559 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:18:35.913536    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40ef677a-3e8c-41aa-8f8e-a2661ed670ec-dshm\") pod \"e2e-same-id-a-9e116000-kserve-7969f5cbf5-ltt6l\" (UID: \"40ef677a-3e8c-41aa-8f8e-a2661ed670ec\") " pod="llm/e2e-same-id-a-9e116000-kserve-7969f5cbf5-ltt6l"
Jul 23 15:18:35.913724 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:18:35.913705    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40ef677a-3e8c-41aa-8f8e-a2661ed670ec-tls-certs\") pod \"e2e-same-id-a-9e116000-kserve-7969f5cbf5-ltt6l\" (UID: \"40ef677a-3e8c-41aa-8f8e-a2661ed670ec\") " pod="llm/e2e-same-id-a-9e116000-kserve-7969f5cbf5-ltt6l"
Jul 23 15:18:35.928677 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:18:35.928648    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztd9z\" (UniqueName: \"kubernetes.io/projected/40ef677a-3e8c-41aa-8f8e-a2661ed670ec-kube-api-access-ztd9z\") pod \"e2e-same-id-a-9e116000-kserve-7969f5cbf5-ltt6l\" (UID: \"40ef677a-3e8c-41aa-8f8e-a2661ed670ec\") " pod="llm/e2e-same-id-a-9e116000-kserve-7969f5cbf5-ltt6l"
Jul 23 15:18:36.056768 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:18:36.056673    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-9e116000-kserve-7969f5cbf5-ltt6l"
Jul 23 15:18:36.204621 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:18:36.204593    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-9e116000-kserve-7969f5cbf5-ltt6l"]
Jul 23 15:18:36.207200 ip-10-0-141-105 kubenswrapper[2560]: W0723 15:18:36.207173    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod40ef677a_3e8c_41aa_8f8e_a2661ed670ec.slice/crio-a0e5f753757491d73a771460b5fe2d545feab1629a43183df8bd11cc6b6692db WatchSource:0}: Error finding container a0e5f753757491d73a771460b5fe2d545feab1629a43183df8bd11cc6b6692db: Status 404 returned error can't find the container with id a0e5f753757491d73a771460b5fe2d545feab1629a43183df8bd11cc6b6692db
Jul 23 15:18:36.743515 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:18:36.743478    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-9e116000-kserve-7969f5cbf5-ltt6l" event={"ID":"40ef677a-3e8c-41aa-8f8e-a2661ed670ec","Type":"ContainerStarted","Data":"d1275c8ee5292f24fd1598f99017121071529ead5e7a5089eb42a64070752d67"}
Jul 23 15:18:36.743515 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:18:36.743522    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-9e116000-kserve-7969f5cbf5-ltt6l" event={"ID":"40ef677a-3e8c-41aa-8f8e-a2661ed670ec","Type":"ContainerStarted","Data":"a0e5f753757491d73a771460b5fe2d545feab1629a43183df8bd11cc6b6692db"}
Jul 23 15:18:36.763507 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:18:36.763455    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-9e116000-kserve-7969f5cbf5-ltt6l" podStartSLOduration=1.763439958 podStartE2EDuration="1.763439958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 15:18:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 15:18:36.762093622 +0000 UTC m=+1558.269967371" watchObservedRunningTime="2026-07-23 15:18:36.763439958 +0000 UTC m=+1558.271313705"
Jul 23 15:18:46.057399 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:18:46.057307    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-9e116000-kserve-7969f5cbf5-ltt6l"
Jul 23 15:18:46.057399 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:18:46.057363    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-9e116000-kserve-7969f5cbf5-ltt6l"
Jul 23 15:18:46.070624 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:18:46.070596    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-9e116000-kserve-7969f5cbf5-ltt6l"
Jul 23 15:18:46.806629 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:18:46.806602    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-9e116000-kserve-7969f5cbf5-ltt6l"
Jul 23 15:18:48.017426 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:18:48.017388    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cgwsg"]
Jul 23 15:18:56.130903 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:18:56.130863    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-9e116000-kserve-5c99658655-qntp2"]
Jul 23 15:18:56.134454 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:18:56.134433    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-9e116000-kserve-5c99658655-qntp2"
Jul 23 15:18:56.137449 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:18:56.137430    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-9e116000-kserve-self-signed-certs\""
Jul 23 15:18:56.144323 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:18:56.144298    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-9e116000-kserve-5c99658655-qntp2"]
Jul 23 15:18:56.194686 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:18:56.194653    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b29ba9b0-f1fb-4b64-94bc-29d67b0581f4-home\") pod \"e2e-same-id-b-9e116000-kserve-5c99658655-qntp2\" (UID: \"b29ba9b0-f1fb-4b64-94bc-29d67b0581f4\") " pod="llm/e2e-same-id-b-9e116000-kserve-5c99658655-qntp2"
Jul 23 15:18:56.194846 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:18:56.194717    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b29ba9b0-f1fb-4b64-94bc-29d67b0581f4-model-cache\") pod \"e2e-same-id-b-9e116000-kserve-5c99658655-qntp2\" (UID: \"b29ba9b0-f1fb-4b64-94bc-29d67b0581f4\") " pod="llm/e2e-same-id-b-9e116000-kserve-5c99658655-qntp2"
Jul 23 15:18:56.194846 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:18:56.194750    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q26mw\" (UniqueName: \"kubernetes.io/projected/b29ba9b0-f1fb-4b64-94bc-29d67b0581f4-kube-api-access-q26mw\") pod \"e2e-same-id-b-9e116000-kserve-5c99658655-qntp2\" (UID: \"b29ba9b0-f1fb-4b64-94bc-29d67b0581f4\") " pod="llm/e2e-same-id-b-9e116000-kserve-5c99658655-qntp2"
Jul 23 15:18:56.194846 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:18:56.194803    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b29ba9b0-f1fb-4b64-94bc-29d67b0581f4-tmp-dir\") pod \"e2e-same-id-b-9e116000-kserve-5c99658655-qntp2\" (UID: \"b29ba9b0-f1fb-4b64-94bc-29d67b0581f4\") " pod="llm/e2e-same-id-b-9e116000-kserve-5c99658655-qntp2"
Jul 23 15:18:56.194846 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:18:56.194820    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b29ba9b0-f1fb-4b64-94bc-29d67b0581f4-tls-certs\") pod \"e2e-same-id-b-9e116000-kserve-5c99658655-qntp2\" (UID: \"b29ba9b0-f1fb-4b64-94bc-29d67b0581f4\") " pod="llm/e2e-same-id-b-9e116000-kserve-5c99658655-qntp2"
Jul 23 15:18:56.194979 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:18:56.194864    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b29ba9b0-f1fb-4b64-94bc-29d67b0581f4-dshm\") pod \"e2e-same-id-b-9e116000-kserve-5c99658655-qntp2\" (UID: \"b29ba9b0-f1fb-4b64-94bc-29d67b0581f4\") " pod="llm/e2e-same-id-b-9e116000-kserve-5c99658655-qntp2"
Jul 23 15:18:56.296312 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:18:56.296268    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b29ba9b0-f1fb-4b64-94bc-29d67b0581f4-model-cache\") pod \"e2e-same-id-b-9e116000-kserve-5c99658655-qntp2\" (UID: \"b29ba9b0-f1fb-4b64-94bc-29d67b0581f4\") " pod="llm/e2e-same-id-b-9e116000-kserve-5c99658655-qntp2"
Jul 23 15:18:56.296312 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:18:56.296310    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q26mw\" (UniqueName: \"kubernetes.io/projected/b29ba9b0-f1fb-4b64-94bc-29d67b0581f4-kube-api-access-q26mw\") pod \"e2e-same-id-b-9e116000-kserve-5c99658655-qntp2\" (UID: \"b29ba9b0-f1fb-4b64-94bc-29d67b0581f4\") " pod="llm/e2e-same-id-b-9e116000-kserve-5c99658655-qntp2"
Jul 23 15:18:56.296555 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:18:56.296343    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b29ba9b0-f1fb-4b64-94bc-29d67b0581f4-tmp-dir\") pod \"e2e-same-id-b-9e116000-kserve-5c99658655-qntp2\" (UID: \"b29ba9b0-f1fb-4b64-94bc-29d67b0581f4\") " pod="llm/e2e-same-id-b-9e116000-kserve-5c99658655-qntp2"
Jul 23 15:18:56.296555 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:18:56.296357    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b29ba9b0-f1fb-4b64-94bc-29d67b0581f4-tls-certs\") pod \"e2e-same-id-b-9e116000-kserve-5c99658655-qntp2\" (UID: \"b29ba9b0-f1fb-4b64-94bc-29d67b0581f4\") " pod="llm/e2e-same-id-b-9e116000-kserve-5c99658655-qntp2"
Jul 23 15:18:56.296555 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:18:56.296405    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b29ba9b0-f1fb-4b64-94bc-29d67b0581f4-dshm\") pod \"e2e-same-id-b-9e116000-kserve-5c99658655-qntp2\" (UID: \"b29ba9b0-f1fb-4b64-94bc-29d67b0581f4\") " pod="llm/e2e-same-id-b-9e116000-kserve-5c99658655-qntp2"
Jul 23 15:18:56.296555 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:18:56.296462    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b29ba9b0-f1fb-4b64-94bc-29d67b0581f4-home\") pod \"e2e-same-id-b-9e116000-kserve-5c99658655-qntp2\" (UID: \"b29ba9b0-f1fb-4b64-94bc-29d67b0581f4\") " pod="llm/e2e-same-id-b-9e116000-kserve-5c99658655-qntp2"
Jul 23 15:18:56.296760 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:18:56.296731    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b29ba9b0-f1fb-4b64-94bc-29d67b0581f4-model-cache\") pod \"e2e-same-id-b-9e116000-kserve-5c99658655-qntp2\" (UID: \"b29ba9b0-f1fb-4b64-94bc-29d67b0581f4\") " pod="llm/e2e-same-id-b-9e116000-kserve-5c99658655-qntp2"
Jul 23 15:18:56.296804 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:18:56.296761    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b29ba9b0-f1fb-4b64-94bc-29d67b0581f4-tmp-dir\") pod \"e2e-same-id-b-9e116000-kserve-5c99658655-qntp2\" (UID: \"b29ba9b0-f1fb-4b64-94bc-29d67b0581f4\") " pod="llm/e2e-same-id-b-9e116000-kserve-5c99658655-qntp2"
Jul 23 15:18:56.296804 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:18:56.296799    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b29ba9b0-f1fb-4b64-94bc-29d67b0581f4-home\") pod \"e2e-same-id-b-9e116000-kserve-5c99658655-qntp2\" (UID: \"b29ba9b0-f1fb-4b64-94bc-29d67b0581f4\") " pod="llm/e2e-same-id-b-9e116000-kserve-5c99658655-qntp2"
Jul 23 15:18:56.298898 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:18:56.298876    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b29ba9b0-f1fb-4b64-94bc-29d67b0581f4-dshm\") pod \"e2e-same-id-b-9e116000-kserve-5c99658655-qntp2\" (UID: \"b29ba9b0-f1fb-4b64-94bc-29d67b0581f4\") " pod="llm/e2e-same-id-b-9e116000-kserve-5c99658655-qntp2"
Jul 23 15:18:56.299181 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:18:56.299160    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b29ba9b0-f1fb-4b64-94bc-29d67b0581f4-tls-certs\") pod \"e2e-same-id-b-9e116000-kserve-5c99658655-qntp2\" (UID: \"b29ba9b0-f1fb-4b64-94bc-29d67b0581f4\") " pod="llm/e2e-same-id-b-9e116000-kserve-5c99658655-qntp2"
Jul 23 15:18:56.305385 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:18:56.305361    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q26mw\" (UniqueName: \"kubernetes.io/projected/b29ba9b0-f1fb-4b64-94bc-29d67b0581f4-kube-api-access-q26mw\") pod \"e2e-same-id-b-9e116000-kserve-5c99658655-qntp2\" (UID: \"b29ba9b0-f1fb-4b64-94bc-29d67b0581f4\") " pod="llm/e2e-same-id-b-9e116000-kserve-5c99658655-qntp2"
Jul 23 15:18:56.445346 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:18:56.445220    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-9e116000-kserve-5c99658655-qntp2"
Jul 23 15:18:56.585161 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:18:56.585126    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-9e116000-kserve-5c99658655-qntp2"]
Jul 23 15:18:56.588858 ip-10-0-141-105 kubenswrapper[2560]: W0723 15:18:56.588831    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb29ba9b0_f1fb_4b64_94bc_29d67b0581f4.slice/crio-01673ca0b541b21471edc9a3c5b18b1a18e2feaea53d7bbadb137e73235873b6 WatchSource:0}: Error finding container 01673ca0b541b21471edc9a3c5b18b1a18e2feaea53d7bbadb137e73235873b6: Status 404 returned error can't find the container with id 01673ca0b541b21471edc9a3c5b18b1a18e2feaea53d7bbadb137e73235873b6
Jul 23 15:18:56.842647 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:18:56.842613    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-9e116000-kserve-5c99658655-qntp2" event={"ID":"b29ba9b0-f1fb-4b64-94bc-29d67b0581f4","Type":"ContainerStarted","Data":"18a1d51012ebf0f1eea25b532cb4d0f0f93de1406df9c3225a520e24cafe4b92"}
Jul 23 15:18:56.842647 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:18:56.842653    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-9e116000-kserve-5c99658655-qntp2" event={"ID":"b29ba9b0-f1fb-4b64-94bc-29d67b0581f4","Type":"ContainerStarted","Data":"01673ca0b541b21471edc9a3c5b18b1a18e2feaea53d7bbadb137e73235873b6"}
Jul 23 15:18:56.861200 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:18:56.861142    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-9e116000-kserve-5c99658655-qntp2" podStartSLOduration=0.861124599 podStartE2EDuration="861.124599ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 15:18:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 15:18:56.860091872 +0000 UTC m=+1578.367965619" watchObservedRunningTime="2026-07-23 15:18:56.861124599 +0000 UTC m=+1578.368998397"
Jul 23 15:19:00.627908 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:19:00.627863    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cgwsg"]
Jul 23 15:19:06.446215 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:19:06.446169    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-9e116000-kserve-5c99658655-qntp2"
Jul 23 15:19:06.446641 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:19:06.446257    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-9e116000-kserve-5c99658655-qntp2"
Jul 23 15:19:06.458857 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:19:06.458826    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-9e116000-kserve-5c99658655-qntp2"
Jul 23 15:19:06.901914 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:19:06.901883    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-9e116000-kserve-5c99658655-qntp2"
Jul 23 15:19:37.144834 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:19:37.144740    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-9e116000-kserve-7969f5cbf5-ltt6l"]
Jul 23 15:19:38.638960 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:19:38.638927    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-9e116000-kserve-5c99658655-qntp2"]
Jul 23 15:19:41.126060 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:19:41.126012    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cgwsg"]
Jul 23 15:19:52.166122 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:19:52.166065    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-9e116000-kserve-7969f5cbf5-ltt6l" podUID="40ef677a-3e8c-41aa-8f8e-a2661ed670ec" containerName="main" containerID="cri-o://d1275c8ee5292f24fd1598f99017121071529ead5e7a5089eb42a64070752d67" gracePeriod=45
Jul 23 15:19:52.422142 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:19:52.422076    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-9e116000-kserve-7969f5cbf5-ltt6l"
Jul 23 15:19:52.510900 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:19:52.510857    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ztd9z\" (UniqueName: \"kubernetes.io/projected/40ef677a-3e8c-41aa-8f8e-a2661ed670ec-kube-api-access-ztd9z\") pod \"40ef677a-3e8c-41aa-8f8e-a2661ed670ec\" (UID: \"40ef677a-3e8c-41aa-8f8e-a2661ed670ec\") "
Jul 23 15:19:52.510900 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:19:52.510907    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40ef677a-3e8c-41aa-8f8e-a2661ed670ec-tls-certs\") pod \"40ef677a-3e8c-41aa-8f8e-a2661ed670ec\" (UID: \"40ef677a-3e8c-41aa-8f8e-a2661ed670ec\") "
Jul 23 15:19:52.511152 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:19:52.510948    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40ef677a-3e8c-41aa-8f8e-a2661ed670ec-dshm\") pod \"40ef677a-3e8c-41aa-8f8e-a2661ed670ec\" (UID: \"40ef677a-3e8c-41aa-8f8e-a2661ed670ec\") "
Jul 23 15:19:52.511152 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:19:52.510988    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40ef677a-3e8c-41aa-8f8e-a2661ed670ec-home\") pod \"40ef677a-3e8c-41aa-8f8e-a2661ed670ec\" (UID: \"40ef677a-3e8c-41aa-8f8e-a2661ed670ec\") "
Jul 23 15:19:52.511152 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:19:52.511035    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40ef677a-3e8c-41aa-8f8e-a2661ed670ec-model-cache\") pod \"40ef677a-3e8c-41aa-8f8e-a2661ed670ec\" (UID: \"40ef677a-3e8c-41aa-8f8e-a2661ed670ec\") "
Jul 23 15:19:52.511152 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:19:52.511079    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40ef677a-3e8c-41aa-8f8e-a2661ed670ec-tmp-dir\") pod \"40ef677a-3e8c-41aa-8f8e-a2661ed670ec\" (UID: \"40ef677a-3e8c-41aa-8f8e-a2661ed670ec\") "
Jul 23 15:19:52.511377 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:19:52.511296    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40ef677a-3e8c-41aa-8f8e-a2661ed670ec-home" (OuterVolumeSpecName: "home") pod "40ef677a-3e8c-41aa-8f8e-a2661ed670ec" (UID: "40ef677a-3e8c-41aa-8f8e-a2661ed670ec"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 15:19:52.511377 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:19:52.511333    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40ef677a-3e8c-41aa-8f8e-a2661ed670ec-model-cache" (OuterVolumeSpecName: "model-cache") pod "40ef677a-3e8c-41aa-8f8e-a2661ed670ec" (UID: "40ef677a-3e8c-41aa-8f8e-a2661ed670ec"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 15:19:52.511493 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:19:52.511422    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40ef677a-3e8c-41aa-8f8e-a2661ed670ec-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "40ef677a-3e8c-41aa-8f8e-a2661ed670ec" (UID: "40ef677a-3e8c-41aa-8f8e-a2661ed670ec"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 15:19:52.511493 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:19:52.511433    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40ef677a-3e8c-41aa-8f8e-a2661ed670ec-home\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Jul 23 15:19:52.511493 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:19:52.511459    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40ef677a-3e8c-41aa-8f8e-a2661ed670ec-model-cache\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Jul 23 15:19:52.513399 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:19:52.513377    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40ef677a-3e8c-41aa-8f8e-a2661ed670ec-dshm" (OuterVolumeSpecName: "dshm") pod "40ef677a-3e8c-41aa-8f8e-a2661ed670ec" (UID: "40ef677a-3e8c-41aa-8f8e-a2661ed670ec"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 15:19:52.513466 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:19:52.513379    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40ef677a-3e8c-41aa-8f8e-a2661ed670ec-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "40ef677a-3e8c-41aa-8f8e-a2661ed670ec" (UID: "40ef677a-3e8c-41aa-8f8e-a2661ed670ec"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 15:19:52.513510 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:19:52.513479    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40ef677a-3e8c-41aa-8f8e-a2661ed670ec-kube-api-access-ztd9z" (OuterVolumeSpecName: "kube-api-access-ztd9z") pod "40ef677a-3e8c-41aa-8f8e-a2661ed670ec" (UID: "40ef677a-3e8c-41aa-8f8e-a2661ed670ec"). InnerVolumeSpecName "kube-api-access-ztd9z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 15:19:52.612023 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:19:52.611984    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztd9z\" (UniqueName: \"kubernetes.io/projected/40ef677a-3e8c-41aa-8f8e-a2661ed670ec-kube-api-access-ztd9z\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Jul 23 15:19:52.612023 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:19:52.612018    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40ef677a-3e8c-41aa-8f8e-a2661ed670ec-tls-certs\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Jul 23 15:19:52.612023 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:19:52.612030    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40ef677a-3e8c-41aa-8f8e-a2661ed670ec-dshm\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Jul 23 15:19:52.612375 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:19:52.612040    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40ef677a-3e8c-41aa-8f8e-a2661ed670ec-tmp-dir\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Jul 23 15:19:53.119755 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:19:53.119714    2560 generic.go:356] "Generic (PLEG): container finished" podID="40ef677a-3e8c-41aa-8f8e-a2661ed670ec" containerID="d1275c8ee5292f24fd1598f99017121071529ead5e7a5089eb42a64070752d67" exitCode=0
Jul 23 15:19:53.119924 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:19:53.119780    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-9e116000-kserve-7969f5cbf5-ltt6l" event={"ID":"40ef677a-3e8c-41aa-8f8e-a2661ed670ec","Type":"ContainerDied","Data":"d1275c8ee5292f24fd1598f99017121071529ead5e7a5089eb42a64070752d67"}
Jul 23 15:19:53.119924 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:19:53.119824    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-9e116000-kserve-7969f5cbf5-ltt6l" event={"ID":"40ef677a-3e8c-41aa-8f8e-a2661ed670ec","Type":"ContainerDied","Data":"a0e5f753757491d73a771460b5fe2d545feab1629a43183df8bd11cc6b6692db"}
Jul 23 15:19:53.119924 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:19:53.119786    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-9e116000-kserve-7969f5cbf5-ltt6l"
Jul 23 15:19:53.119924 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:19:53.119876    2560 scope.go:117] "RemoveContainer" containerID="d1275c8ee5292f24fd1598f99017121071529ead5e7a5089eb42a64070752d67"
Jul 23 15:19:53.130811 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:19:53.130789    2560 scope.go:117] "RemoveContainer" containerID="d1275c8ee5292f24fd1598f99017121071529ead5e7a5089eb42a64070752d67"
Jul 23 15:19:53.131165 ip-10-0-141-105 kubenswrapper[2560]: E0723 15:19:53.131145    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1275c8ee5292f24fd1598f99017121071529ead5e7a5089eb42a64070752d67\": container with ID starting with d1275c8ee5292f24fd1598f99017121071529ead5e7a5089eb42a64070752d67 not found: ID does not exist" containerID="d1275c8ee5292f24fd1598f99017121071529ead5e7a5089eb42a64070752d67"
Jul 23 15:19:53.131213 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:19:53.131176    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1275c8ee5292f24fd1598f99017121071529ead5e7a5089eb42a64070752d67"} err="failed to get container status \"d1275c8ee5292f24fd1598f99017121071529ead5e7a5089eb42a64070752d67\": rpc error: code = NotFound desc = could not find container \"d1275c8ee5292f24fd1598f99017121071529ead5e7a5089eb42a64070752d67\": container with ID starting with d1275c8ee5292f24fd1598f99017121071529ead5e7a5089eb42a64070752d67 not found: ID does not exist"
Jul 23 15:19:53.136041 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:19:53.136001    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-9e116000-kserve-7969f5cbf5-ltt6l"]
Jul 23 15:19:53.138300 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:19:53.138271    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-9e116000-kserve-7969f5cbf5-ltt6l"]
Jul 23 15:19:53.662655 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:19:53.662614    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-9e116000-kserve-5c99658655-qntp2" podUID="b29ba9b0-f1fb-4b64-94bc-29d67b0581f4" containerName="main" containerID="cri-o://18a1d51012ebf0f1eea25b532cb4d0f0f93de1406df9c3225a520e24cafe4b92" gracePeriod=45
Jul 23 15:19:53.903188 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:19:53.903160    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-9e116000-kserve-5c99658655-qntp2"
Jul 23 15:19:54.026486 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:19:54.026453    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b29ba9b0-f1fb-4b64-94bc-29d67b0581f4-dshm\") pod \"b29ba9b0-f1fb-4b64-94bc-29d67b0581f4\" (UID: \"b29ba9b0-f1fb-4b64-94bc-29d67b0581f4\") "
Jul 23 15:19:54.026486 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:19:54.026496    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b29ba9b0-f1fb-4b64-94bc-29d67b0581f4-model-cache\") pod \"b29ba9b0-f1fb-4b64-94bc-29d67b0581f4\" (UID: \"b29ba9b0-f1fb-4b64-94bc-29d67b0581f4\") "
Jul 23 15:19:54.026755 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:19:54.026563    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b29ba9b0-f1fb-4b64-94bc-29d67b0581f4-tmp-dir\") pod \"b29ba9b0-f1fb-4b64-94bc-29d67b0581f4\" (UID: \"b29ba9b0-f1fb-4b64-94bc-29d67b0581f4\") "
Jul 23 15:19:54.026755 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:19:54.026580    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b29ba9b0-f1fb-4b64-94bc-29d67b0581f4-tls-certs\") pod \"b29ba9b0-f1fb-4b64-94bc-29d67b0581f4\" (UID: \"b29ba9b0-f1fb-4b64-94bc-29d67b0581f4\") "
Jul 23 15:19:54.026755 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:19:54.026636    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b29ba9b0-f1fb-4b64-94bc-29d67b0581f4-home\") pod \"b29ba9b0-f1fb-4b64-94bc-29d67b0581f4\" (UID: \"b29ba9b0-f1fb-4b64-94bc-29d67b0581f4\") "
Jul 23 15:19:54.026755 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:19:54.026658    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q26mw\" (UniqueName: \"kubernetes.io/projected/b29ba9b0-f1fb-4b64-94bc-29d67b0581f4-kube-api-access-q26mw\") pod \"b29ba9b0-f1fb-4b64-94bc-29d67b0581f4\" (UID: \"b29ba9b0-f1fb-4b64-94bc-29d67b0581f4\") "
Jul 23 15:19:54.026962 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:19:54.026805    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b29ba9b0-f1fb-4b64-94bc-29d67b0581f4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b29ba9b0-f1fb-4b64-94bc-29d67b0581f4" (UID: "b29ba9b0-f1fb-4b64-94bc-29d67b0581f4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 15:19:54.026962 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:19:54.026834    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b29ba9b0-f1fb-4b64-94bc-29d67b0581f4-model-cache" (OuterVolumeSpecName: "model-cache") pod "b29ba9b0-f1fb-4b64-94bc-29d67b0581f4" (UID: "b29ba9b0-f1fb-4b64-94bc-29d67b0581f4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 15:19:54.026962 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:19:54.026915    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b29ba9b0-f1fb-4b64-94bc-29d67b0581f4-home" (OuterVolumeSpecName: "home") pod "b29ba9b0-f1fb-4b64-94bc-29d67b0581f4" (UID: "b29ba9b0-f1fb-4b64-94bc-29d67b0581f4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 15:19:54.026962 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:19:54.026958    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b29ba9b0-f1fb-4b64-94bc-29d67b0581f4-model-cache\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Jul 23 15:19:54.027153 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:19:54.026988    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b29ba9b0-f1fb-4b64-94bc-29d67b0581f4-tmp-dir\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Jul 23 15:19:54.028764 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:19:54.028733    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b29ba9b0-f1fb-4b64-94bc-29d67b0581f4-dshm" (OuterVolumeSpecName: "dshm") pod "b29ba9b0-f1fb-4b64-94bc-29d67b0581f4" (UID: "b29ba9b0-f1fb-4b64-94bc-29d67b0581f4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 15:19:54.028869 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:19:54.028837    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b29ba9b0-f1fb-4b64-94bc-29d67b0581f4-kube-api-access-q26mw" (OuterVolumeSpecName: "kube-api-access-q26mw") pod "b29ba9b0-f1fb-4b64-94bc-29d67b0581f4" (UID: "b29ba9b0-f1fb-4b64-94bc-29d67b0581f4"). InnerVolumeSpecName "kube-api-access-q26mw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 15:19:54.028907 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:19:54.028877    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b29ba9b0-f1fb-4b64-94bc-29d67b0581f4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b29ba9b0-f1fb-4b64-94bc-29d67b0581f4" (UID: "b29ba9b0-f1fb-4b64-94bc-29d67b0581f4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 15:19:54.125814 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:19:54.125771    2560 generic.go:356] "Generic (PLEG): container finished" podID="b29ba9b0-f1fb-4b64-94bc-29d67b0581f4" containerID="18a1d51012ebf0f1eea25b532cb4d0f0f93de1406df9c3225a520e24cafe4b92" exitCode=0
Jul 23 15:19:54.126007 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:19:54.125821    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-9e116000-kserve-5c99658655-qntp2"
Jul 23 15:19:54.126007 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:19:54.125862    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-9e116000-kserve-5c99658655-qntp2" event={"ID":"b29ba9b0-f1fb-4b64-94bc-29d67b0581f4","Type":"ContainerDied","Data":"18a1d51012ebf0f1eea25b532cb4d0f0f93de1406df9c3225a520e24cafe4b92"}
Jul 23 15:19:54.126007 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:19:54.125897    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-9e116000-kserve-5c99658655-qntp2" event={"ID":"b29ba9b0-f1fb-4b64-94bc-29d67b0581f4","Type":"ContainerDied","Data":"01673ca0b541b21471edc9a3c5b18b1a18e2feaea53d7bbadb137e73235873b6"}
Jul 23 15:19:54.126007 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:19:54.125914    2560 scope.go:117] "RemoveContainer" containerID="18a1d51012ebf0f1eea25b532cb4d0f0f93de1406df9c3225a520e24cafe4b92"
Jul 23 15:19:54.127519 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:19:54.127500    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b29ba9b0-f1fb-4b64-94bc-29d67b0581f4-home\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Jul 23 15:19:54.127636 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:19:54.127522    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q26mw\" (UniqueName: \"kubernetes.io/projected/b29ba9b0-f1fb-4b64-94bc-29d67b0581f4-kube-api-access-q26mw\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Jul 23 15:19:54.127636 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:19:54.127534    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b29ba9b0-f1fb-4b64-94bc-29d67b0581f4-dshm\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Jul 23 15:19:54.127636 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:19:54.127542    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b29ba9b0-f1fb-4b64-94bc-29d67b0581f4-tls-certs\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Jul 23 15:19:54.135016 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:19:54.134999    2560 scope.go:117] "RemoveContainer" containerID="18a1d51012ebf0f1eea25b532cb4d0f0f93de1406df9c3225a520e24cafe4b92"
Jul 23 15:19:54.135301 ip-10-0-141-105 kubenswrapper[2560]: E0723 15:19:54.135280    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18a1d51012ebf0f1eea25b532cb4d0f0f93de1406df9c3225a520e24cafe4b92\": container with ID starting with 18a1d51012ebf0f1eea25b532cb4d0f0f93de1406df9c3225a520e24cafe4b92 not found: ID does not exist" containerID="18a1d51012ebf0f1eea25b532cb4d0f0f93de1406df9c3225a520e24cafe4b92"
Jul 23 15:19:54.135374 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:19:54.135313    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18a1d51012ebf0f1eea25b532cb4d0f0f93de1406df9c3225a520e24cafe4b92"} err="failed to get container status \"18a1d51012ebf0f1eea25b532cb4d0f0f93de1406df9c3225a520e24cafe4b92\": rpc error: code = NotFound desc = could not find container \"18a1d51012ebf0f1eea25b532cb4d0f0f93de1406df9c3225a520e24cafe4b92\": container with ID starting with 18a1d51012ebf0f1eea25b532cb4d0f0f93de1406df9c3225a520e24cafe4b92 not found: ID does not exist"
Jul 23 15:19:54.147354 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:19:54.147324    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-9e116000-kserve-5c99658655-qntp2"]
Jul 23 15:19:54.152342 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:19:54.152314    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-9e116000-kserve-5c99658655-qntp2"]
Jul 23 15:19:54.819462 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:19:54.819411    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cgwsg"]
Jul 23 15:19:55.062667 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:19:55.062632    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="40ef677a-3e8c-41aa-8f8e-a2661ed670ec" path="/var/lib/kubelet/pods/40ef677a-3e8c-41aa-8f8e-a2661ed670ec/volumes"
Jul 23 15:19:55.063016 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:19:55.063003    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b29ba9b0-f1fb-4b64-94bc-29d67b0581f4" path="/var/lib/kubelet/pods/b29ba9b0-f1fb-4b64-94bc-29d67b0581f4/volumes"
Jul 23 15:20:04.231644 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:20:04.231588    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cgwsg"]
Jul 23 15:20:16.424259 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:20:16.424210    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cgwsg"]
Jul 23 15:20:27.626109 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:20:27.626075    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cgwsg"]
Jul 23 15:20:37.228628 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:20:37.228598    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cgwsg"]
Jul 23 15:20:48.035829 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:20:48.035786    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cgwsg"]
Jul 23 15:20:57.424138 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:20:57.424104    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cgwsg"]
Jul 23 15:21:07.020393 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:21:07.020306    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cgwsg"]
Jul 23 15:21:20.719866 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:21:20.719833    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cgwsg"]
Jul 23 15:21:29.922211 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:21:29.922168    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cgwsg"]
Jul 23 15:21:39.330495 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:21:39.330462    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cgwsg"]
Jul 23 15:21:48.822553 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:21:48.822514    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cgwsg"]
Jul 23 15:22:08.430214 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:22:08.430177    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cgwsg"]
Jul 23 15:22:19.530656 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:22:19.530616    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cgwsg"]
Jul 23 15:22:31.427645 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:22:31.427558    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cgwsg"]
Jul 23 15:22:40.253841 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:22:40.253729    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z4fvn_e0c66f26-06fe-4273-ba29-fa20ce89b873/ovn-acl-logging/0.log"
Jul 23 15:22:40.261548 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:22:40.257915    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z4fvn_e0c66f26-06fe-4273-ba29-fa20ce89b873/ovn-acl-logging/0.log"
Jul 23 15:22:44.122150 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:22:44.122110    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cgwsg"]
Jul 23 15:22:57.827074 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:22:57.827039    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cgwsg"]
Jul 23 15:23:07.321245 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:23:07.321196    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cgwsg"]
Jul 23 15:23:17.133996 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:23:17.133954    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cgwsg"]
Jul 23 15:23:20.279674 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:23:20.279636    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-af2a71ee-6c856d49bc-8qr7z"]
Jul 23 15:23:20.280074 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:23:20.279993    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40ef677a-3e8c-41aa-8f8e-a2661ed670ec" containerName="main"
Jul 23 15:23:20.280074 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:23:20.280004    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="40ef677a-3e8c-41aa-8f8e-a2661ed670ec" containerName="main"
Jul 23 15:23:20.280074 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:23:20.280020    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b29ba9b0-f1fb-4b64-94bc-29d67b0581f4" containerName="main"
Jul 23 15:23:20.280074 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:23:20.280025    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="b29ba9b0-f1fb-4b64-94bc-29d67b0581f4" containerName="main"
Jul 23 15:23:20.280264 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:23:20.280088    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="b29ba9b0-f1fb-4b64-94bc-29d67b0581f4" containerName="main"
Jul 23 15:23:20.280264 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:23:20.280100    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="40ef677a-3e8c-41aa-8f8e-a2661ed670ec" containerName="main"
Jul 23 15:23:20.283309 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:23:20.283291    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-af2a71ee-6c856d49bc-8qr7z"
Jul 23 15:23:20.286341 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:23:20.286320    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-af2a71ee-serving-cert\""
Jul 23 15:23:20.286556 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:23:20.286542    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-rdm8l\""
Jul 23 15:23:20.286612 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:23:20.286598    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 23 15:23:20.287908 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:23:20.287889    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 23 15:23:20.299057 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:23:20.299027    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-af2a71ee-6c856d49bc-8qr7z"]
Jul 23 15:23:20.463821 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:23:20.463781    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0998ecea-3691-4339-ae1e-58f633b6153e-openshift-service-ca\") pod \"maas-api-e2e-ait-af2a71ee-6c856d49bc-8qr7z\" (UID: \"0998ecea-3691-4339-ae1e-58f633b6153e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-af2a71ee-6c856d49bc-8qr7z"
Jul 23 15:23:20.463821 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:23:20.463820    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9mr6\" (UniqueName: \"kubernetes.io/projected/0998ecea-3691-4339-ae1e-58f633b6153e-kube-api-access-t9mr6\") pod \"maas-api-e2e-ait-af2a71ee-6c856d49bc-8qr7z\" (UID: \"0998ecea-3691-4339-ae1e-58f633b6153e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-af2a71ee-6c856d49bc-8qr7z"
Jul 23 15:23:20.464094 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:23:20.463860    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0998ecea-3691-4339-ae1e-58f633b6153e-maas-api-tls\") pod \"maas-api-e2e-ait-af2a71ee-6c856d49bc-8qr7z\" (UID: \"0998ecea-3691-4339-ae1e-58f633b6153e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-af2a71ee-6c856d49bc-8qr7z"
Jul 23 15:23:20.565407 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:23:20.565312    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0998ecea-3691-4339-ae1e-58f633b6153e-maas-api-tls\") pod \"maas-api-e2e-ait-af2a71ee-6c856d49bc-8qr7z\" (UID: \"0998ecea-3691-4339-ae1e-58f633b6153e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-af2a71ee-6c856d49bc-8qr7z"
Jul 23 15:23:20.565563 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:23:20.565468    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0998ecea-3691-4339-ae1e-58f633b6153e-openshift-service-ca\") pod \"maas-api-e2e-ait-af2a71ee-6c856d49bc-8qr7z\" (UID: \"0998ecea-3691-4339-ae1e-58f633b6153e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-af2a71ee-6c856d49bc-8qr7z"
Jul 23 15:23:20.565563 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:23:20.565499    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9mr6\" (UniqueName: \"kubernetes.io/projected/0998ecea-3691-4339-ae1e-58f633b6153e-kube-api-access-t9mr6\") pod \"maas-api-e2e-ait-af2a71ee-6c856d49bc-8qr7z\" (UID: \"0998ecea-3691-4339-ae1e-58f633b6153e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-af2a71ee-6c856d49bc-8qr7z"
Jul 23 15:23:20.566095 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:23:20.566076    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0998ecea-3691-4339-ae1e-58f633b6153e-openshift-service-ca\") pod \"maas-api-e2e-ait-af2a71ee-6c856d49bc-8qr7z\" (UID: \"0998ecea-3691-4339-ae1e-58f633b6153e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-af2a71ee-6c856d49bc-8qr7z"
Jul 23 15:23:20.568045 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:23:20.568024    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0998ecea-3691-4339-ae1e-58f633b6153e-maas-api-tls\") pod \"maas-api-e2e-ait-af2a71ee-6c856d49bc-8qr7z\" (UID: \"0998ecea-3691-4339-ae1e-58f633b6153e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-af2a71ee-6c856d49bc-8qr7z"
Jul 23 15:23:20.574800 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:23:20.574762    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9mr6\" (UniqueName: \"kubernetes.io/projected/0998ecea-3691-4339-ae1e-58f633b6153e-kube-api-access-t9mr6\") pod \"maas-api-e2e-ait-af2a71ee-6c856d49bc-8qr7z\" (UID: \"0998ecea-3691-4339-ae1e-58f633b6153e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-af2a71ee-6c856d49bc-8qr7z"
Jul 23 15:23:20.594667 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:23:20.594638    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-af2a71ee-6c856d49bc-8qr7z"
Jul 23 15:23:20.730694 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:23:20.730666    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-af2a71ee-6c856d49bc-8qr7z"]
Jul 23 15:23:20.732791 ip-10-0-141-105 kubenswrapper[2560]: W0723 15:23:20.732761    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0998ecea_3691_4339_ae1e_58f633b6153e.slice/crio-2662223e7e732d76dbf5f62bfa0057a559cf55e91ff227d8d7cf421a11a68792 WatchSource:0}: Error finding container 2662223e7e732d76dbf5f62bfa0057a559cf55e91ff227d8d7cf421a11a68792: Status 404 returned error can't find the container with id 2662223e7e732d76dbf5f62bfa0057a559cf55e91ff227d8d7cf421a11a68792
Jul 23 15:23:20.734079 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:23:20.734059    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 23 15:23:21.088138 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:23:21.088094    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-af2a71ee-6c856d49bc-8qr7z" event={"ID":"0998ecea-3691-4339-ae1e-58f633b6153e","Type":"ContainerStarted","Data":"2662223e7e732d76dbf5f62bfa0057a559cf55e91ff227d8d7cf421a11a68792"}
Jul 23 15:23:24.105320 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:23:24.105269    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-af2a71ee-6c856d49bc-8qr7z" event={"ID":"0998ecea-3691-4339-ae1e-58f633b6153e","Type":"ContainerStarted","Data":"59804b026abec2c9146908f4914274ea3c8483bd2e6b118a6108b72e0ba08187"}
Jul 23 15:23:24.105727 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:23:24.105357    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-af2a71ee-6c856d49bc-8qr7z"
Jul 23 15:23:24.123797 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:23:24.123741    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-af2a71ee-6c856d49bc-8qr7z" podStartSLOduration=1.785295672 podStartE2EDuration="4.123724473s" totalImagesPullingTime="2.338428801s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 15:23:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 15:23:24.122184098 +0000 UTC m=+1845.630057848" watchObservedRunningTime="2026-07-23 15:23:24.123724473 +0000 UTC m=+1845.631598221"
Jul 23 15:23:27.132577 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:23:27.128827    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cgwsg"]
Jul 23 15:23:30.117092 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:23:30.117062    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-af2a71ee-6c856d49bc-8qr7z"
Jul 23 15:23:56.266609 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:23:56.266556    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-af2a71ee-6c856d49bc-8qr7z"]
Jul 23 15:23:56.267121 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:23:56.266827    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-af2a71ee-6c856d49bc-8qr7z" podUID="0998ecea-3691-4339-ae1e-58f633b6153e" containerName="maas-api" containerID="cri-o://59804b026abec2c9146908f4914274ea3c8483bd2e6b118a6108b72e0ba08187" gracePeriod=30
Jul 23 15:23:56.518519 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:23:56.518450    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-af2a71ee-6c856d49bc-8qr7z"
Jul 23 15:23:56.593126 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:23:56.593086    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t9mr6\" (UniqueName: \"kubernetes.io/projected/0998ecea-3691-4339-ae1e-58f633b6153e-kube-api-access-t9mr6\") pod \"0998ecea-3691-4339-ae1e-58f633b6153e\" (UID: \"0998ecea-3691-4339-ae1e-58f633b6153e\") "
Jul 23 15:23:56.593126 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:23:56.593132    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0998ecea-3691-4339-ae1e-58f633b6153e-openshift-service-ca\") pod \"0998ecea-3691-4339-ae1e-58f633b6153e\" (UID: \"0998ecea-3691-4339-ae1e-58f633b6153e\") "
Jul 23 15:23:56.593427 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:23:56.593194    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0998ecea-3691-4339-ae1e-58f633b6153e-maas-api-tls\") pod \"0998ecea-3691-4339-ae1e-58f633b6153e\" (UID: \"0998ecea-3691-4339-ae1e-58f633b6153e\") "
Jul 23 15:23:56.593551 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:23:56.593524    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0998ecea-3691-4339-ae1e-58f633b6153e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0998ecea-3691-4339-ae1e-58f633b6153e" (UID: "0998ecea-3691-4339-ae1e-58f633b6153e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 15:23:56.595560 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:23:56.595519    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0998ecea-3691-4339-ae1e-58f633b6153e-kube-api-access-t9mr6" (OuterVolumeSpecName: "kube-api-access-t9mr6") pod "0998ecea-3691-4339-ae1e-58f633b6153e" (UID: "0998ecea-3691-4339-ae1e-58f633b6153e"). InnerVolumeSpecName "kube-api-access-t9mr6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 15:23:56.595677 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:23:56.595607    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0998ecea-3691-4339-ae1e-58f633b6153e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0998ecea-3691-4339-ae1e-58f633b6153e" (UID: "0998ecea-3691-4339-ae1e-58f633b6153e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 15:23:56.694723 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:23:56.694683    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9mr6\" (UniqueName: \"kubernetes.io/projected/0998ecea-3691-4339-ae1e-58f633b6153e-kube-api-access-t9mr6\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Jul 23 15:23:56.694723 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:23:56.694715    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0998ecea-3691-4339-ae1e-58f633b6153e-openshift-service-ca\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Jul 23 15:23:56.694723 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:23:56.694726    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0998ecea-3691-4339-ae1e-58f633b6153e-maas-api-tls\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Jul 23 15:23:57.258493 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:23:57.258457    2560 generic.go:356] "Generic (PLEG): container finished" podID="0998ecea-3691-4339-ae1e-58f633b6153e" containerID="59804b026abec2c9146908f4914274ea3c8483bd2e6b118a6108b72e0ba08187" exitCode=0
Jul 23 15:23:57.258675 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:23:57.258504    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-af2a71ee-6c856d49bc-8qr7z"
Jul 23 15:23:57.258675 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:23:57.258524    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-af2a71ee-6c856d49bc-8qr7z" event={"ID":"0998ecea-3691-4339-ae1e-58f633b6153e","Type":"ContainerDied","Data":"59804b026abec2c9146908f4914274ea3c8483bd2e6b118a6108b72e0ba08187"}
Jul 23 15:23:57.258675 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:23:57.258578    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-af2a71ee-6c856d49bc-8qr7z" event={"ID":"0998ecea-3691-4339-ae1e-58f633b6153e","Type":"ContainerDied","Data":"2662223e7e732d76dbf5f62bfa0057a559cf55e91ff227d8d7cf421a11a68792"}
Jul 23 15:23:57.258675 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:23:57.258611    2560 scope.go:117] "RemoveContainer" containerID="59804b026abec2c9146908f4914274ea3c8483bd2e6b118a6108b72e0ba08187"
Jul 23 15:23:57.267767 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:23:57.267585    2560 scope.go:117] "RemoveContainer" containerID="59804b026abec2c9146908f4914274ea3c8483bd2e6b118a6108b72e0ba08187"
Jul 23 15:23:57.268037 ip-10-0-141-105 kubenswrapper[2560]: E0723 15:23:57.267926    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59804b026abec2c9146908f4914274ea3c8483bd2e6b118a6108b72e0ba08187\": container with ID starting with 59804b026abec2c9146908f4914274ea3c8483bd2e6b118a6108b72e0ba08187 not found: ID does not exist" containerID="59804b026abec2c9146908f4914274ea3c8483bd2e6b118a6108b72e0ba08187"
Jul 23 15:23:57.268037 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:23:57.267952    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59804b026abec2c9146908f4914274ea3c8483bd2e6b118a6108b72e0ba08187"} err="failed to get container status \"59804b026abec2c9146908f4914274ea3c8483bd2e6b118a6108b72e0ba08187\": rpc error: code = NotFound desc = could not find container \"59804b026abec2c9146908f4914274ea3c8483bd2e6b118a6108b72e0ba08187\": container with ID starting with 59804b026abec2c9146908f4914274ea3c8483bd2e6b118a6108b72e0ba08187 not found: ID does not exist"
Jul 23 15:23:57.276587 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:23:57.276558    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-af2a71ee-6c856d49bc-8qr7z"]
Jul 23 15:23:57.280677 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:23:57.280652    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-af2a71ee-6c856d49bc-8qr7z"]
Jul 23 15:23:59.024176 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:23:59.024133    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cgwsg"]
Jul 23 15:23:59.062657 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:23:59.062624    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0998ecea-3691-4339-ae1e-58f633b6153e" path="/var/lib/kubelet/pods/0998ecea-3691-4339-ae1e-58f633b6153e/volumes"
Jul 23 15:24:04.771811 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:24:04.771773    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-e35a5c33-fc6587f58-65bjs"]
Jul 23 15:24:04.772312 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:24:04.772110    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0998ecea-3691-4339-ae1e-58f633b6153e" containerName="maas-api"
Jul 23 15:24:04.772312 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:24:04.772121    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="0998ecea-3691-4339-ae1e-58f633b6153e" containerName="maas-api"
Jul 23 15:24:04.772312 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:24:04.772193    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="0998ecea-3691-4339-ae1e-58f633b6153e" containerName="maas-api"
Jul 23 15:24:04.776718 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:24:04.776695    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e35a5c33-fc6587f58-65bjs"
Jul 23 15:24:04.780536 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:24:04.780510    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-rdm8l\""
Jul 23 15:24:04.780685 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:24:04.780538    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-e35a5c33-serving-cert\""
Jul 23 15:24:04.781059 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:24:04.781043    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 23 15:24:04.783520 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:24:04.783252    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 23 15:24:04.795857 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:24:04.795831    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-e35a5c33-fc6587f58-65bjs"]
Jul 23 15:24:04.868274 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:24:04.868207    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce78fddc-012b-4be7-a17b-4fd076da0938-openshift-service-ca\") pod \"maas-api-e2e-migrate-e35a5c33-fc6587f58-65bjs\" (UID: \"ce78fddc-012b-4be7-a17b-4fd076da0938\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e35a5c33-fc6587f58-65bjs"
Jul 23 15:24:04.868493 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:24:04.868364    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce78fddc-012b-4be7-a17b-4fd076da0938-maas-api-tls\") pod \"maas-api-e2e-migrate-e35a5c33-fc6587f58-65bjs\" (UID: \"ce78fddc-012b-4be7-a17b-4fd076da0938\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e35a5c33-fc6587f58-65bjs"
Jul 23 15:24:04.868564 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:24:04.868491    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84bn7\" (UniqueName: \"kubernetes.io/projected/ce78fddc-012b-4be7-a17b-4fd076da0938-kube-api-access-84bn7\") pod \"maas-api-e2e-migrate-e35a5c33-fc6587f58-65bjs\" (UID: \"ce78fddc-012b-4be7-a17b-4fd076da0938\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e35a5c33-fc6587f58-65bjs"
Jul 23 15:24:04.969597 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:24:04.969551    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce78fddc-012b-4be7-a17b-4fd076da0938-maas-api-tls\") pod \"maas-api-e2e-migrate-e35a5c33-fc6587f58-65bjs\" (UID: \"ce78fddc-012b-4be7-a17b-4fd076da0938\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e35a5c33-fc6587f58-65bjs"
Jul 23 15:24:04.969802 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:24:04.969653    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84bn7\" (UniqueName: \"kubernetes.io/projected/ce78fddc-012b-4be7-a17b-4fd076da0938-kube-api-access-84bn7\") pod \"maas-api-e2e-migrate-e35a5c33-fc6587f58-65bjs\" (UID: \"ce78fddc-012b-4be7-a17b-4fd076da0938\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e35a5c33-fc6587f58-65bjs"
Jul 23 15:24:04.969802 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:24:04.969690    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce78fddc-012b-4be7-a17b-4fd076da0938-openshift-service-ca\") pod \"maas-api-e2e-migrate-e35a5c33-fc6587f58-65bjs\" (UID: \"ce78fddc-012b-4be7-a17b-4fd076da0938\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e35a5c33-fc6587f58-65bjs"
Jul 23 15:24:04.970389 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:24:04.970357    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce78fddc-012b-4be7-a17b-4fd076da0938-openshift-service-ca\") pod \"maas-api-e2e-migrate-e35a5c33-fc6587f58-65bjs\" (UID: \"ce78fddc-012b-4be7-a17b-4fd076da0938\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e35a5c33-fc6587f58-65bjs"
Jul 23 15:24:04.972336 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:24:04.972314    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce78fddc-012b-4be7-a17b-4fd076da0938-maas-api-tls\") pod \"maas-api-e2e-migrate-e35a5c33-fc6587f58-65bjs\" (UID: \"ce78fddc-012b-4be7-a17b-4fd076da0938\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e35a5c33-fc6587f58-65bjs"
Jul 23 15:24:04.979064 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:24:04.979028    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-84bn7\" (UniqueName: \"kubernetes.io/projected/ce78fddc-012b-4be7-a17b-4fd076da0938-kube-api-access-84bn7\") pod \"maas-api-e2e-migrate-e35a5c33-fc6587f58-65bjs\" (UID: \"ce78fddc-012b-4be7-a17b-4fd076da0938\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e35a5c33-fc6587f58-65bjs"
Jul 23 15:24:05.087361 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:24:05.087261    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e35a5c33-fc6587f58-65bjs"
Jul 23 15:24:05.228504 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:24:05.228477    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-e35a5c33-fc6587f58-65bjs"]
Jul 23 15:24:05.231281 ip-10-0-141-105 kubenswrapper[2560]: W0723 15:24:05.231246    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce78fddc_012b_4be7_a17b_4fd076da0938.slice/crio-e51f22e4cffa29c46e7f70095a8059a727d458607b41a6e9a3d751b521e069b0 WatchSource:0}: Error finding container e51f22e4cffa29c46e7f70095a8059a727d458607b41a6e9a3d751b521e069b0: Status 404 returned error can't find the container with id e51f22e4cffa29c46e7f70095a8059a727d458607b41a6e9a3d751b521e069b0
Jul 23 15:24:05.296894 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:24:05.296853    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e35a5c33-fc6587f58-65bjs" event={"ID":"ce78fddc-012b-4be7-a17b-4fd076da0938","Type":"ContainerStarted","Data":"e51f22e4cffa29c46e7f70095a8059a727d458607b41a6e9a3d751b521e069b0"}
Jul 23 15:24:06.302930 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:24:06.302894    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e35a5c33-fc6587f58-65bjs" event={"ID":"ce78fddc-012b-4be7-a17b-4fd076da0938","Type":"ContainerStarted","Data":"e15df128d10ac8a7fbb8b54e9858c2648e505b5ecc172e5e4e52555edd9c000f"}
Jul 23 15:24:06.303357 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:24:06.302975    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e35a5c33-fc6587f58-65bjs"
Jul 23 15:24:06.320908 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:24:06.320851    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e35a5c33-fc6587f58-65bjs" podStartSLOduration=1.84415977 podStartE2EDuration="2.320833671s" totalImagesPullingTime="476.673901ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 15:24:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 15:24:06.320051488 +0000 UTC m=+1887.827925236" watchObservedRunningTime="2026-07-23 15:24:06.320833671 +0000 UTC m=+1887.828707419"
Jul 23 15:24:08.326909 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:24:08.326877    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cgwsg"]
Jul 23 15:24:12.314089 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:24:12.314058    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e35a5c33-fc6587f58-65bjs"
Jul 23 15:24:40.015502 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:24:40.015467    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-e35a5c33-fc6587f58-65bjs"]
Jul 23 15:24:40.016018 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:24:40.015747    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e35a5c33-fc6587f58-65bjs" podUID="ce78fddc-012b-4be7-a17b-4fd076da0938" containerName="maas-api" containerID="cri-o://e15df128d10ac8a7fbb8b54e9858c2648e505b5ecc172e5e4e52555edd9c000f" gracePeriod=30
Jul 23 15:24:40.262737 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:24:40.262707    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e35a5c33-fc6587f58-65bjs"
Jul 23 15:24:40.378535 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:24:40.378432    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce78fddc-012b-4be7-a17b-4fd076da0938-openshift-service-ca\") pod \"ce78fddc-012b-4be7-a17b-4fd076da0938\" (UID: \"ce78fddc-012b-4be7-a17b-4fd076da0938\") "
Jul 23 15:24:40.378535 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:24:40.378527    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce78fddc-012b-4be7-a17b-4fd076da0938-maas-api-tls\") pod \"ce78fddc-012b-4be7-a17b-4fd076da0938\" (UID: \"ce78fddc-012b-4be7-a17b-4fd076da0938\") "
Jul 23 15:24:40.378729 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:24:40.378565    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-84bn7\" (UniqueName: \"kubernetes.io/projected/ce78fddc-012b-4be7-a17b-4fd076da0938-kube-api-access-84bn7\") pod \"ce78fddc-012b-4be7-a17b-4fd076da0938\" (UID: \"ce78fddc-012b-4be7-a17b-4fd076da0938\") "
Jul 23 15:24:40.378803 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:24:40.378776    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce78fddc-012b-4be7-a17b-4fd076da0938-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ce78fddc-012b-4be7-a17b-4fd076da0938" (UID: "ce78fddc-012b-4be7-a17b-4fd076da0938"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 15:24:40.380919 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:24:40.380893    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce78fddc-012b-4be7-a17b-4fd076da0938-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ce78fddc-012b-4be7-a17b-4fd076da0938" (UID: "ce78fddc-012b-4be7-a17b-4fd076da0938"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 15:24:40.380919 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:24:40.380901    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce78fddc-012b-4be7-a17b-4fd076da0938-kube-api-access-84bn7" (OuterVolumeSpecName: "kube-api-access-84bn7") pod "ce78fddc-012b-4be7-a17b-4fd076da0938" (UID: "ce78fddc-012b-4be7-a17b-4fd076da0938"). InnerVolumeSpecName "kube-api-access-84bn7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 15:24:40.454128 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:24:40.454094    2560 generic.go:356] "Generic (PLEG): container finished" podID="ce78fddc-012b-4be7-a17b-4fd076da0938" containerID="e15df128d10ac8a7fbb8b54e9858c2648e505b5ecc172e5e4e52555edd9c000f" exitCode=0
Jul 23 15:24:40.454316 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:24:40.454152    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e35a5c33-fc6587f58-65bjs"
Jul 23 15:24:40.454316 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:24:40.454169    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e35a5c33-fc6587f58-65bjs" event={"ID":"ce78fddc-012b-4be7-a17b-4fd076da0938","Type":"ContainerDied","Data":"e15df128d10ac8a7fbb8b54e9858c2648e505b5ecc172e5e4e52555edd9c000f"}
Jul 23 15:24:40.454316 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:24:40.454211    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e35a5c33-fc6587f58-65bjs" event={"ID":"ce78fddc-012b-4be7-a17b-4fd076da0938","Type":"ContainerDied","Data":"e51f22e4cffa29c46e7f70095a8059a727d458607b41a6e9a3d751b521e069b0"}
Jul 23 15:24:40.454316 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:24:40.454252    2560 scope.go:117] "RemoveContainer" containerID="e15df128d10ac8a7fbb8b54e9858c2648e505b5ecc172e5e4e52555edd9c000f"
Jul 23 15:24:40.463507 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:24:40.463484    2560 scope.go:117] "RemoveContainer" containerID="e15df128d10ac8a7fbb8b54e9858c2648e505b5ecc172e5e4e52555edd9c000f"
Jul 23 15:24:40.463806 ip-10-0-141-105 kubenswrapper[2560]: E0723 15:24:40.463786    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e15df128d10ac8a7fbb8b54e9858c2648e505b5ecc172e5e4e52555edd9c000f\": container with ID starting with e15df128d10ac8a7fbb8b54e9858c2648e505b5ecc172e5e4e52555edd9c000f not found: ID does not exist" containerID="e15df128d10ac8a7fbb8b54e9858c2648e505b5ecc172e5e4e52555edd9c000f"
Jul 23 15:24:40.463856 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:24:40.463831    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e15df128d10ac8a7fbb8b54e9858c2648e505b5ecc172e5e4e52555edd9c000f"} err="failed to get container status \"e15df128d10ac8a7fbb8b54e9858c2648e505b5ecc172e5e4e52555edd9c000f\": rpc error: code = NotFound desc = could not find container \"e15df128d10ac8a7fbb8b54e9858c2648e505b5ecc172e5e4e52555edd9c000f\": container with ID starting with e15df128d10ac8a7fbb8b54e9858c2648e505b5ecc172e5e4e52555edd9c000f not found: ID does not exist"
Jul 23 15:24:40.475680 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:24:40.475654    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-e35a5c33-fc6587f58-65bjs"]
Jul 23 15:24:40.479797 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:24:40.479776    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce78fddc-012b-4be7-a17b-4fd076da0938-openshift-service-ca\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Jul 23 15:24:40.479797 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:24:40.479797    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce78fddc-012b-4be7-a17b-4fd076da0938-maas-api-tls\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Jul 23 15:24:40.479943 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:24:40.479807    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84bn7\" (UniqueName: \"kubernetes.io/projected/ce78fddc-012b-4be7-a17b-4fd076da0938-kube-api-access-84bn7\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Jul 23 15:24:40.480143 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:24:40.480125    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-e35a5c33-fc6587f58-65bjs"]
Jul 23 15:24:41.062318 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:24:41.062286    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ce78fddc-012b-4be7-a17b-4fd076da0938" path="/var/lib/kubelet/pods/ce78fddc-012b-4be7-a17b-4fd076da0938/volumes"
Jul 23 15:24:42.821906 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:24:42.821869    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cgwsg"]
Jul 23 15:24:48.729014 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:24:48.728976    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ee7b3491-84896475c-npdxp"]
Jul 23 15:24:48.729521 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:24:48.729357    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce78fddc-012b-4be7-a17b-4fd076da0938" containerName="maas-api"
Jul 23 15:24:48.729521 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:24:48.729369    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce78fddc-012b-4be7-a17b-4fd076da0938" containerName="maas-api"
Jul 23 15:24:48.729521 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:24:48.729438    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce78fddc-012b-4be7-a17b-4fd076da0938" containerName="maas-api"
Jul 23 15:24:48.733629 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:24:48.733611    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ee7b3491-84896475c-npdxp"
Jul 23 15:24:48.736533 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:24:48.736511    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 23 15:24:48.736533 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:24:48.736520    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-rdm8l\""
Jul 23 15:24:48.736742 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:24:48.736517    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 23 15:24:48.737698 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:24:48.737679    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-ee7b3491-serving-cert\""
Jul 23 15:24:48.741470 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:24:48.741443    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ee7b3491-84896475c-npdxp"]
Jul 23 15:24:48.859556 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:24:48.859504    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1593b1f0-6528-4f1a-b5f6-7fe844134df7-maas-api-tls\") pod \"maas-api-e2e-ait-ee7b3491-84896475c-npdxp\" (UID: \"1593b1f0-6528-4f1a-b5f6-7fe844134df7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ee7b3491-84896475c-npdxp"
Jul 23 15:24:48.859745 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:24:48.859576    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwchx\" (UniqueName: \"kubernetes.io/projected/1593b1f0-6528-4f1a-b5f6-7fe844134df7-kube-api-access-jwchx\") pod \"maas-api-e2e-ait-ee7b3491-84896475c-npdxp\" (UID: \"1593b1f0-6528-4f1a-b5f6-7fe844134df7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ee7b3491-84896475c-npdxp"
Jul 23 15:24:48.859745 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:24:48.859654    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1593b1f0-6528-4f1a-b5f6-7fe844134df7-openshift-service-ca\") pod \"maas-api-e2e-ait-ee7b3491-84896475c-npdxp\" (UID: \"1593b1f0-6528-4f1a-b5f6-7fe844134df7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ee7b3491-84896475c-npdxp"
Jul 23 15:24:48.960883 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:24:48.960850    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1593b1f0-6528-4f1a-b5f6-7fe844134df7-openshift-service-ca\") pod \"maas-api-e2e-ait-ee7b3491-84896475c-npdxp\" (UID: \"1593b1f0-6528-4f1a-b5f6-7fe844134df7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ee7b3491-84896475c-npdxp"
Jul 23 15:24:48.961069 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:24:48.960902    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1593b1f0-6528-4f1a-b5f6-7fe844134df7-maas-api-tls\") pod \"maas-api-e2e-ait-ee7b3491-84896475c-npdxp\" (UID: \"1593b1f0-6528-4f1a-b5f6-7fe844134df7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ee7b3491-84896475c-npdxp"
Jul 23 15:24:48.961069 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:24:48.960931    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwchx\" (UniqueName: \"kubernetes.io/projected/1593b1f0-6528-4f1a-b5f6-7fe844134df7-kube-api-access-jwchx\") pod \"maas-api-e2e-ait-ee7b3491-84896475c-npdxp\" (UID: \"1593b1f0-6528-4f1a-b5f6-7fe844134df7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ee7b3491-84896475c-npdxp"
Jul 23 15:24:48.961546 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:24:48.961512    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1593b1f0-6528-4f1a-b5f6-7fe844134df7-openshift-service-ca\") pod \"maas-api-e2e-ait-ee7b3491-84896475c-npdxp\" (UID: \"1593b1f0-6528-4f1a-b5f6-7fe844134df7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ee7b3491-84896475c-npdxp"
Jul 23 15:24:48.963573 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:24:48.963553    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1593b1f0-6528-4f1a-b5f6-7fe844134df7-maas-api-tls\") pod \"maas-api-e2e-ait-ee7b3491-84896475c-npdxp\" (UID: \"1593b1f0-6528-4f1a-b5f6-7fe844134df7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ee7b3491-84896475c-npdxp"
Jul 23 15:24:48.969192 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:24:48.969169    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwchx\" (UniqueName: \"kubernetes.io/projected/1593b1f0-6528-4f1a-b5f6-7fe844134df7-kube-api-access-jwchx\") pod \"maas-api-e2e-ait-ee7b3491-84896475c-npdxp\" (UID: \"1593b1f0-6528-4f1a-b5f6-7fe844134df7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ee7b3491-84896475c-npdxp"
Jul 23 15:24:49.045088 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:24:49.044991    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ee7b3491-84896475c-npdxp"
Jul 23 15:24:49.185844 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:24:49.185824    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ee7b3491-84896475c-npdxp"]
Jul 23 15:24:49.188759 ip-10-0-141-105 kubenswrapper[2560]: W0723 15:24:49.188725    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1593b1f0_6528_4f1a_b5f6_7fe844134df7.slice/crio-7e5b1d71fddaa7e551c6b8719cd00c481af6cffe78a3821ada6d3c16d771e9d7 WatchSource:0}: Error finding container 7e5b1d71fddaa7e551c6b8719cd00c481af6cffe78a3821ada6d3c16d771e9d7: Status 404 returned error can't find the container with id 7e5b1d71fddaa7e551c6b8719cd00c481af6cffe78a3821ada6d3c16d771e9d7
Jul 23 15:24:49.497946 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:24:49.497910    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ee7b3491-84896475c-npdxp" event={"ID":"1593b1f0-6528-4f1a-b5f6-7fe844134df7","Type":"ContainerStarted","Data":"7e5b1d71fddaa7e551c6b8719cd00c481af6cffe78a3821ada6d3c16d771e9d7"}
Jul 23 15:24:51.514729 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:24:51.514687    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ee7b3491-84896475c-npdxp" event={"ID":"1593b1f0-6528-4f1a-b5f6-7fe844134df7","Type":"ContainerStarted","Data":"7db11c00093765c5ed2079f1ddc2b2eaf2c6ac73b691baf7d75db3f6c5662361"}
Jul 23 15:24:51.515151 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:24:51.514877    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ee7b3491-84896475c-npdxp"
Jul 23 15:24:51.533054 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:24:51.533004    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ee7b3491-84896475c-npdxp" podStartSLOduration=1.681741148 podStartE2EDuration="3.532984339s" totalImagesPullingTime="1.851243191s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 15:24:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 15:24:51.530730416 +0000 UTC m=+1933.038604164" watchObservedRunningTime="2026-07-23 15:24:51.532984339 +0000 UTC m=+1933.040858087"
Jul 23 15:24:54.428621 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:24:54.428585    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cgwsg"]
Jul 23 15:24:57.527048 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:24:57.527010    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ee7b3491-84896475c-npdxp"
Jul 23 15:25:25.464017 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:25:25.463981    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ee7b3491-84896475c-npdxp"]
Jul 23 15:25:25.464571 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:25:25.464189    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ee7b3491-84896475c-npdxp" podUID="1593b1f0-6528-4f1a-b5f6-7fe844134df7" containerName="maas-api" containerID="cri-o://7db11c00093765c5ed2079f1ddc2b2eaf2c6ac73b691baf7d75db3f6c5662361" gracePeriod=30
Jul 23 15:25:25.675325 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:25:25.675284    2560 generic.go:356] "Generic (PLEG): container finished" podID="1593b1f0-6528-4f1a-b5f6-7fe844134df7" containerID="7db11c00093765c5ed2079f1ddc2b2eaf2c6ac73b691baf7d75db3f6c5662361" exitCode=0
Jul 23 15:25:25.675509 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:25:25.675370    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ee7b3491-84896475c-npdxp" event={"ID":"1593b1f0-6528-4f1a-b5f6-7fe844134df7","Type":"ContainerDied","Data":"7db11c00093765c5ed2079f1ddc2b2eaf2c6ac73b691baf7d75db3f6c5662361"}
Jul 23 15:25:25.714648 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:25:25.714627    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ee7b3491-84896475c-npdxp"
Jul 23 15:25:25.780697 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:25:25.780663    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jwchx\" (UniqueName: \"kubernetes.io/projected/1593b1f0-6528-4f1a-b5f6-7fe844134df7-kube-api-access-jwchx\") pod \"1593b1f0-6528-4f1a-b5f6-7fe844134df7\" (UID: \"1593b1f0-6528-4f1a-b5f6-7fe844134df7\") "
Jul 23 15:25:25.780892 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:25:25.780711    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1593b1f0-6528-4f1a-b5f6-7fe844134df7-openshift-service-ca\") pod \"1593b1f0-6528-4f1a-b5f6-7fe844134df7\" (UID: \"1593b1f0-6528-4f1a-b5f6-7fe844134df7\") "
Jul 23 15:25:25.780892 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:25:25.780743    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1593b1f0-6528-4f1a-b5f6-7fe844134df7-maas-api-tls\") pod \"1593b1f0-6528-4f1a-b5f6-7fe844134df7\" (UID: \"1593b1f0-6528-4f1a-b5f6-7fe844134df7\") "
Jul 23 15:25:25.781055 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:25:25.781028    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1593b1f0-6528-4f1a-b5f6-7fe844134df7-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1593b1f0-6528-4f1a-b5f6-7fe844134df7" (UID: "1593b1f0-6528-4f1a-b5f6-7fe844134df7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 15:25:25.782927 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:25:25.782898    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1593b1f0-6528-4f1a-b5f6-7fe844134df7-kube-api-access-jwchx" (OuterVolumeSpecName: "kube-api-access-jwchx") pod "1593b1f0-6528-4f1a-b5f6-7fe844134df7" (UID: "1593b1f0-6528-4f1a-b5f6-7fe844134df7"). InnerVolumeSpecName "kube-api-access-jwchx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 15:25:25.783041 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:25:25.782971    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1593b1f0-6528-4f1a-b5f6-7fe844134df7-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1593b1f0-6528-4f1a-b5f6-7fe844134df7" (UID: "1593b1f0-6528-4f1a-b5f6-7fe844134df7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 15:25:25.881640 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:25:25.881599    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwchx\" (UniqueName: \"kubernetes.io/projected/1593b1f0-6528-4f1a-b5f6-7fe844134df7-kube-api-access-jwchx\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Jul 23 15:25:25.881640 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:25:25.881632    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1593b1f0-6528-4f1a-b5f6-7fe844134df7-openshift-service-ca\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Jul 23 15:25:25.881640 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:25:25.881646    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1593b1f0-6528-4f1a-b5f6-7fe844134df7-maas-api-tls\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Jul 23 15:25:26.680746 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:25:26.680718    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ee7b3491-84896475c-npdxp"
Jul 23 15:25:26.681142 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:25:26.680717    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ee7b3491-84896475c-npdxp" event={"ID":"1593b1f0-6528-4f1a-b5f6-7fe844134df7","Type":"ContainerDied","Data":"7e5b1d71fddaa7e551c6b8719cd00c481af6cffe78a3821ada6d3c16d771e9d7"}
Jul 23 15:25:26.681142 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:25:26.680860    2560 scope.go:117] "RemoveContainer" containerID="7db11c00093765c5ed2079f1ddc2b2eaf2c6ac73b691baf7d75db3f6c5662361"
Jul 23 15:25:26.701906 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:25:26.701879    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ee7b3491-84896475c-npdxp"]
Jul 23 15:25:26.707719 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:25:26.707693    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ee7b3491-84896475c-npdxp"]
Jul 23 15:25:27.063007 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:25:27.062966    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1593b1f0-6528-4f1a-b5f6-7fe844134df7" path="/var/lib/kubelet/pods/1593b1f0-6528-4f1a-b5f6-7fe844134df7/volumes"
Jul 23 15:25:27.521797 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:25:27.521761    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cgwsg"]
Jul 23 15:25:34.036332 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:25:34.036251    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-3e83224e-7d989ddf98-fg9ws"]
Jul 23 15:25:34.036799 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:25:34.036635    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1593b1f0-6528-4f1a-b5f6-7fe844134df7" containerName="maas-api"
Jul 23 15:25:34.036799 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:25:34.036648    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="1593b1f0-6528-4f1a-b5f6-7fe844134df7" containerName="maas-api"
Jul 23 15:25:34.036799 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:25:34.036702    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="1593b1f0-6528-4f1a-b5f6-7fe844134df7" containerName="maas-api"
Jul 23 15:25:34.041037 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:25:34.041014    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3e83224e-7d989ddf98-fg9ws"
Jul 23 15:25:34.043872 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:25:34.043845    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 23 15:25:34.044023 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:25:34.043896    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 23 15:25:34.044023 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:25:34.043904    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-3e83224e-serving-cert\""
Jul 23 15:25:34.045160 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:25:34.045140    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-rdm8l\""
Jul 23 15:25:34.049913 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:25:34.049889    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-3e83224e-7d989ddf98-fg9ws"]
Jul 23 15:25:34.160185 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:25:34.160150    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvlpl\" (UniqueName: \"kubernetes.io/projected/541382a7-c632-4398-a06b-eeee01a89970-kube-api-access-fvlpl\") pod \"maas-api-e2e-derive-3e83224e-7d989ddf98-fg9ws\" (UID: \"541382a7-c632-4398-a06b-eeee01a89970\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3e83224e-7d989ddf98-fg9ws"
Jul 23 15:25:34.160384 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:25:34.160210    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/541382a7-c632-4398-a06b-eeee01a89970-maas-api-tls\") pod \"maas-api-e2e-derive-3e83224e-7d989ddf98-fg9ws\" (UID: \"541382a7-c632-4398-a06b-eeee01a89970\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3e83224e-7d989ddf98-fg9ws"
Jul 23 15:25:34.160384 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:25:34.160323    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/541382a7-c632-4398-a06b-eeee01a89970-openshift-service-ca\") pod \"maas-api-e2e-derive-3e83224e-7d989ddf98-fg9ws\" (UID: \"541382a7-c632-4398-a06b-eeee01a89970\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3e83224e-7d989ddf98-fg9ws"
Jul 23 15:25:34.261466 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:25:34.261428    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvlpl\" (UniqueName: \"kubernetes.io/projected/541382a7-c632-4398-a06b-eeee01a89970-kube-api-access-fvlpl\") pod \"maas-api-e2e-derive-3e83224e-7d989ddf98-fg9ws\" (UID: \"541382a7-c632-4398-a06b-eeee01a89970\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3e83224e-7d989ddf98-fg9ws"
Jul 23 15:25:34.261639 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:25:34.261493    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/541382a7-c632-4398-a06b-eeee01a89970-maas-api-tls\") pod \"maas-api-e2e-derive-3e83224e-7d989ddf98-fg9ws\" (UID: \"541382a7-c632-4398-a06b-eeee01a89970\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3e83224e-7d989ddf98-fg9ws"
Jul 23 15:25:34.261639 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:25:34.261534    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/541382a7-c632-4398-a06b-eeee01a89970-openshift-service-ca\") pod \"maas-api-e2e-derive-3e83224e-7d989ddf98-fg9ws\" (UID: \"541382a7-c632-4398-a06b-eeee01a89970\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3e83224e-7d989ddf98-fg9ws"
Jul 23 15:25:34.261756 ip-10-0-141-105 kubenswrapper[2560]: E0723 15:25:34.261647    2560 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-3e83224e-serving-cert: secret "maas-api-e2e-derive-3e83224e-serving-cert" not found
Jul 23 15:25:34.261756 ip-10-0-141-105 kubenswrapper[2560]: E0723 15:25:34.261715    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/541382a7-c632-4398-a06b-eeee01a89970-maas-api-tls podName:541382a7-c632-4398-a06b-eeee01a89970 nodeName:}" failed. No retries permitted until 2026-07-23 15:25:34.761697467 +0000 UTC m=+1976.269571192 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/541382a7-c632-4398-a06b-eeee01a89970-maas-api-tls") pod "maas-api-e2e-derive-3e83224e-7d989ddf98-fg9ws" (UID: "541382a7-c632-4398-a06b-eeee01a89970") : secret "maas-api-e2e-derive-3e83224e-serving-cert" not found
Jul 23 15:25:34.262100 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:25:34.262082    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/541382a7-c632-4398-a06b-eeee01a89970-openshift-service-ca\") pod \"maas-api-e2e-derive-3e83224e-7d989ddf98-fg9ws\" (UID: \"541382a7-c632-4398-a06b-eeee01a89970\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3e83224e-7d989ddf98-fg9ws"
Jul 23 15:25:34.270176 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:25:34.270146    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvlpl\" (UniqueName: \"kubernetes.io/projected/541382a7-c632-4398-a06b-eeee01a89970-kube-api-access-fvlpl\") pod \"maas-api-e2e-derive-3e83224e-7d989ddf98-fg9ws\" (UID: \"541382a7-c632-4398-a06b-eeee01a89970\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3e83224e-7d989ddf98-fg9ws"
Jul 23 15:25:34.766976 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:25:34.766942    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/541382a7-c632-4398-a06b-eeee01a89970-maas-api-tls\") pod \"maas-api-e2e-derive-3e83224e-7d989ddf98-fg9ws\" (UID: \"541382a7-c632-4398-a06b-eeee01a89970\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3e83224e-7d989ddf98-fg9ws"
Jul 23 15:25:34.769644 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:25:34.769610    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/541382a7-c632-4398-a06b-eeee01a89970-maas-api-tls\") pod \"maas-api-e2e-derive-3e83224e-7d989ddf98-fg9ws\" (UID: \"541382a7-c632-4398-a06b-eeee01a89970\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3e83224e-7d989ddf98-fg9ws"
Jul 23 15:25:34.954627 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:25:34.954592    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3e83224e-7d989ddf98-fg9ws"
Jul 23 15:25:35.091926 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:25:35.091896    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-3e83224e-7d989ddf98-fg9ws"]
Jul 23 15:25:35.094733 ip-10-0-141-105 kubenswrapper[2560]: W0723 15:25:35.094706    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod541382a7_c632_4398_a06b_eeee01a89970.slice/crio-f2cf2d0e19c37cbb11dae8f635f92c34f6820fd6bd4679ee4a312a7d3df3e204 WatchSource:0}: Error finding container f2cf2d0e19c37cbb11dae8f635f92c34f6820fd6bd4679ee4a312a7d3df3e204: Status 404 returned error can't find the container with id f2cf2d0e19c37cbb11dae8f635f92c34f6820fd6bd4679ee4a312a7d3df3e204
Jul 23 15:25:35.727131 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:25:35.727090    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3e83224e-7d989ddf98-fg9ws" event={"ID":"541382a7-c632-4398-a06b-eeee01a89970","Type":"ContainerStarted","Data":"e6e3c2429df0abd69223b476b86d1a495df4260c001085bf92b70eee1c87a034"}
Jul 23 15:25:35.727334 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:25:35.727195    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3e83224e-7d989ddf98-fg9ws"
Jul 23 15:25:35.727334 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:25:35.727214    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3e83224e-7d989ddf98-fg9ws" event={"ID":"541382a7-c632-4398-a06b-eeee01a89970","Type":"ContainerStarted","Data":"f2cf2d0e19c37cbb11dae8f635f92c34f6820fd6bd4679ee4a312a7d3df3e204"}
Jul 23 15:25:35.743281 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:25:35.743149    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3e83224e-7d989ddf98-fg9ws" podStartSLOduration=1.354245188 podStartE2EDuration="1.743131447s" totalImagesPullingTime="388.886259ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 15:25:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 15:25:35.742378497 +0000 UTC m=+1977.250252245" watchObservedRunningTime="2026-07-23 15:25:35.743131447 +0000 UTC m=+1977.251005195"
Jul 23 15:25:36.820495 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:25:36.820461    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cgwsg"]
Jul 23 15:25:41.737447 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:25:41.737416    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3e83224e-7d989ddf98-fg9ws"
Jul 23 15:26:09.174008 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:26:09.173968    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-3e83224e-7d989ddf98-fg9ws"]
Jul 23 15:26:09.175087 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:26:09.174289    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3e83224e-7d989ddf98-fg9ws" podUID="541382a7-c632-4398-a06b-eeee01a89970" containerName="maas-api" containerID="cri-o://e6e3c2429df0abd69223b476b86d1a495df4260c001085bf92b70eee1c87a034" gracePeriod=30
Jul 23 15:26:09.418010 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:26:09.417986    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3e83224e-7d989ddf98-fg9ws"
Jul 23 15:26:09.472117 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:26:09.472092    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fvlpl\" (UniqueName: \"kubernetes.io/projected/541382a7-c632-4398-a06b-eeee01a89970-kube-api-access-fvlpl\") pod \"541382a7-c632-4398-a06b-eeee01a89970\" (UID: \"541382a7-c632-4398-a06b-eeee01a89970\") "
Jul 23 15:26:09.472117 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:26:09.472122    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/541382a7-c632-4398-a06b-eeee01a89970-openshift-service-ca\") pod \"541382a7-c632-4398-a06b-eeee01a89970\" (UID: \"541382a7-c632-4398-a06b-eeee01a89970\") "
Jul 23 15:26:09.472397 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:26:09.472156    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/541382a7-c632-4398-a06b-eeee01a89970-maas-api-tls\") pod \"541382a7-c632-4398-a06b-eeee01a89970\" (UID: \"541382a7-c632-4398-a06b-eeee01a89970\") "
Jul 23 15:26:09.472546 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:26:09.472520    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/541382a7-c632-4398-a06b-eeee01a89970-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "541382a7-c632-4398-a06b-eeee01a89970" (UID: "541382a7-c632-4398-a06b-eeee01a89970"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 15:26:09.474637 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:26:09.474607    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/541382a7-c632-4398-a06b-eeee01a89970-kube-api-access-fvlpl" (OuterVolumeSpecName: "kube-api-access-fvlpl") pod "541382a7-c632-4398-a06b-eeee01a89970" (UID: "541382a7-c632-4398-a06b-eeee01a89970"). InnerVolumeSpecName "kube-api-access-fvlpl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 15:26:09.474764 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:26:09.474673    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/541382a7-c632-4398-a06b-eeee01a89970-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "541382a7-c632-4398-a06b-eeee01a89970" (UID: "541382a7-c632-4398-a06b-eeee01a89970"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 15:26:09.573501 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:26:09.573466    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvlpl\" (UniqueName: \"kubernetes.io/projected/541382a7-c632-4398-a06b-eeee01a89970-kube-api-access-fvlpl\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Jul 23 15:26:09.573501 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:26:09.573495    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/541382a7-c632-4398-a06b-eeee01a89970-openshift-service-ca\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Jul 23 15:26:09.573501 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:26:09.573506    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/541382a7-c632-4398-a06b-eeee01a89970-maas-api-tls\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Jul 23 15:26:09.891741 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:26:09.891658    2560 generic.go:356] "Generic (PLEG): container finished" podID="541382a7-c632-4398-a06b-eeee01a89970" containerID="e6e3c2429df0abd69223b476b86d1a495df4260c001085bf92b70eee1c87a034" exitCode=0
Jul 23 15:26:09.891741 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:26:09.891706    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3e83224e-7d989ddf98-fg9ws"
Jul 23 15:26:09.891741 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:26:09.891713    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3e83224e-7d989ddf98-fg9ws" event={"ID":"541382a7-c632-4398-a06b-eeee01a89970","Type":"ContainerDied","Data":"e6e3c2429df0abd69223b476b86d1a495df4260c001085bf92b70eee1c87a034"}
Jul 23 15:26:09.891968 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:26:09.891756    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3e83224e-7d989ddf98-fg9ws" event={"ID":"541382a7-c632-4398-a06b-eeee01a89970","Type":"ContainerDied","Data":"f2cf2d0e19c37cbb11dae8f635f92c34f6820fd6bd4679ee4a312a7d3df3e204"}
Jul 23 15:26:09.891968 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:26:09.891779    2560 scope.go:117] "RemoveContainer" containerID="e6e3c2429df0abd69223b476b86d1a495df4260c001085bf92b70eee1c87a034"
Jul 23 15:26:09.900550 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:26:09.900528    2560 scope.go:117] "RemoveContainer" containerID="e6e3c2429df0abd69223b476b86d1a495df4260c001085bf92b70eee1c87a034"
Jul 23 15:26:09.900807 ip-10-0-141-105 kubenswrapper[2560]: E0723 15:26:09.900788    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6e3c2429df0abd69223b476b86d1a495df4260c001085bf92b70eee1c87a034\": container with ID starting with e6e3c2429df0abd69223b476b86d1a495df4260c001085bf92b70eee1c87a034 not found: ID does not exist" containerID="e6e3c2429df0abd69223b476b86d1a495df4260c001085bf92b70eee1c87a034"
Jul 23 15:26:09.900855 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:26:09.900818    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6e3c2429df0abd69223b476b86d1a495df4260c001085bf92b70eee1c87a034"} err="failed to get container status \"e6e3c2429df0abd69223b476b86d1a495df4260c001085bf92b70eee1c87a034\": rpc error: code = NotFound desc = could not find container \"e6e3c2429df0abd69223b476b86d1a495df4260c001085bf92b70eee1c87a034\": container with ID starting with e6e3c2429df0abd69223b476b86d1a495df4260c001085bf92b70eee1c87a034 not found: ID does not exist"
Jul 23 15:26:09.913044 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:26:09.913013    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-3e83224e-7d989ddf98-fg9ws"]
Jul 23 15:26:09.917157 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:26:09.917128    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-3e83224e-7d989ddf98-fg9ws"]
Jul 23 15:26:11.062302 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:26:11.062274    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="541382a7-c632-4398-a06b-eeee01a89970" path="/var/lib/kubelet/pods/541382a7-c632-4398-a06b-eeee01a89970/volumes"
Jul 23 15:26:12.131338 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:26:12.131295    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cgwsg"]
Jul 23 15:27:39.923299 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:27:39.923262    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cgwsg"]
Jul 23 15:27:40.282083 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:27:40.281983    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z4fvn_e0c66f26-06fe-4273-ba29-fa20ce89b873/ovn-acl-logging/0.log"
Jul 23 15:27:40.287592 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:27:40.287571    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z4fvn_e0c66f26-06fe-4273-ba29-fa20ce89b873/ovn-acl-logging/0.log"
Jul 23 15:27:49.428891 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:27:49.428858    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cgwsg"]
Jul 23 15:27:56.338876 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:27:56.338839    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7f6db9fe-bff779489-c55vd"]
Jul 23 15:27:56.339414 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:27:56.339395    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="541382a7-c632-4398-a06b-eeee01a89970" containerName="maas-api"
Jul 23 15:27:56.339501 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:27:56.339418    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="541382a7-c632-4398-a06b-eeee01a89970" containerName="maas-api"
Jul 23 15:27:56.339562 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:27:56.339512    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="541382a7-c632-4398-a06b-eeee01a89970" containerName="maas-api"
Jul 23 15:27:56.342456 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:27:56.342432    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7f6db9fe-bff779489-c55vd"
Jul 23 15:27:56.345207 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:27:56.345184    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-7f6db9fe-serving-cert\""
Jul 23 15:27:56.345359 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:27:56.345271    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 23 15:27:56.345525 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:27:56.345498    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-rdm8l\""
Jul 23 15:27:56.346460 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:27:56.346436    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 23 15:27:56.352788 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:27:56.352761    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7f6db9fe-bff779489-c55vd"]
Jul 23 15:27:56.444642 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:27:56.444600    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3691bff-328c-4586-8f3f-4cafd5a8f05c-openshift-service-ca\") pod \"maas-api-e2e-mt-7f6db9fe-bff779489-c55vd\" (UID: \"d3691bff-328c-4586-8f3f-4cafd5a8f05c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7f6db9fe-bff779489-c55vd"
Jul 23 15:27:56.444827 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:27:56.444678    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3691bff-328c-4586-8f3f-4cafd5a8f05c-maas-api-tls\") pod \"maas-api-e2e-mt-7f6db9fe-bff779489-c55vd\" (UID: \"d3691bff-328c-4586-8f3f-4cafd5a8f05c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7f6db9fe-bff779489-c55vd"
Jul 23 15:27:56.444827 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:27:56.444727    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqwfs\" (UniqueName: \"kubernetes.io/projected/d3691bff-328c-4586-8f3f-4cafd5a8f05c-kube-api-access-pqwfs\") pod \"maas-api-e2e-mt-7f6db9fe-bff779489-c55vd\" (UID: \"d3691bff-328c-4586-8f3f-4cafd5a8f05c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7f6db9fe-bff779489-c55vd"
Jul 23 15:27:56.546158 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:27:56.546108    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3691bff-328c-4586-8f3f-4cafd5a8f05c-openshift-service-ca\") pod \"maas-api-e2e-mt-7f6db9fe-bff779489-c55vd\" (UID: \"d3691bff-328c-4586-8f3f-4cafd5a8f05c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7f6db9fe-bff779489-c55vd"
Jul 23 15:27:56.546389 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:27:56.546215    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3691bff-328c-4586-8f3f-4cafd5a8f05c-maas-api-tls\") pod \"maas-api-e2e-mt-7f6db9fe-bff779489-c55vd\" (UID: \"d3691bff-328c-4586-8f3f-4cafd5a8f05c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7f6db9fe-bff779489-c55vd"
Jul 23 15:27:56.546389 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:27:56.546293    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqwfs\" (UniqueName: \"kubernetes.io/projected/d3691bff-328c-4586-8f3f-4cafd5a8f05c-kube-api-access-pqwfs\") pod \"maas-api-e2e-mt-7f6db9fe-bff779489-c55vd\" (UID: \"d3691bff-328c-4586-8f3f-4cafd5a8f05c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7f6db9fe-bff779489-c55vd"
Jul 23 15:27:56.546389 ip-10-0-141-105 kubenswrapper[2560]: E0723 15:27:56.546371    2560 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-7f6db9fe-serving-cert: secret "maas-api-e2e-mt-7f6db9fe-serving-cert" not found
Jul 23 15:27:56.546564 ip-10-0-141-105 kubenswrapper[2560]: E0723 15:27:56.546462    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3691bff-328c-4586-8f3f-4cafd5a8f05c-maas-api-tls podName:d3691bff-328c-4586-8f3f-4cafd5a8f05c nodeName:}" failed. No retries permitted until 2026-07-23 15:27:57.046443661 +0000 UTC m=+2118.554317388 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d3691bff-328c-4586-8f3f-4cafd5a8f05c-maas-api-tls") pod "maas-api-e2e-mt-7f6db9fe-bff779489-c55vd" (UID: "d3691bff-328c-4586-8f3f-4cafd5a8f05c") : secret "maas-api-e2e-mt-7f6db9fe-serving-cert" not found
Jul 23 15:27:56.546875 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:27:56.546852    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3691bff-328c-4586-8f3f-4cafd5a8f05c-openshift-service-ca\") pod \"maas-api-e2e-mt-7f6db9fe-bff779489-c55vd\" (UID: \"d3691bff-328c-4586-8f3f-4cafd5a8f05c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7f6db9fe-bff779489-c55vd"
Jul 23 15:27:56.555356 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:27:56.555331    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqwfs\" (UniqueName: \"kubernetes.io/projected/d3691bff-328c-4586-8f3f-4cafd5a8f05c-kube-api-access-pqwfs\") pod \"maas-api-e2e-mt-7f6db9fe-bff779489-c55vd\" (UID: \"d3691bff-328c-4586-8f3f-4cafd5a8f05c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7f6db9fe-bff779489-c55vd"
Jul 23 15:27:57.052305 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:27:57.052253    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3691bff-328c-4586-8f3f-4cafd5a8f05c-maas-api-tls\") pod \"maas-api-e2e-mt-7f6db9fe-bff779489-c55vd\" (UID: \"d3691bff-328c-4586-8f3f-4cafd5a8f05c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7f6db9fe-bff779489-c55vd"
Jul 23 15:27:57.055122 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:27:57.055085    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3691bff-328c-4586-8f3f-4cafd5a8f05c-maas-api-tls\") pod \"maas-api-e2e-mt-7f6db9fe-bff779489-c55vd\" (UID: \"d3691bff-328c-4586-8f3f-4cafd5a8f05c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7f6db9fe-bff779489-c55vd"
Jul 23 15:27:57.255739 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:27:57.255693    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7f6db9fe-bff779489-c55vd"
Jul 23 15:27:57.422042 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:27:57.422010    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7f6db9fe-bff779489-c55vd"]
Jul 23 15:27:57.425692 ip-10-0-141-105 kubenswrapper[2560]: W0723 15:27:57.425662    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3691bff_328c_4586_8f3f_4cafd5a8f05c.slice/crio-267f204142b27d3958d7f4ebddaa9c82a2b9ed8a60f0b8fe49cf7e29e5774216 WatchSource:0}: Error finding container 267f204142b27d3958d7f4ebddaa9c82a2b9ed8a60f0b8fe49cf7e29e5774216: Status 404 returned error can't find the container with id 267f204142b27d3958d7f4ebddaa9c82a2b9ed8a60f0b8fe49cf7e29e5774216
Jul 23 15:27:58.411605 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:27:58.411565    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7f6db9fe-bff779489-c55vd" event={"ID":"d3691bff-328c-4586-8f3f-4cafd5a8f05c","Type":"ContainerStarted","Data":"32e8768b131b2b8ebf6805ab3e398e546d4e49af4d0822d3b01e02780c6dd049"}
Jul 23 15:27:58.411821 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:27:58.411632    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7f6db9fe-bff779489-c55vd" event={"ID":"d3691bff-328c-4586-8f3f-4cafd5a8f05c","Type":"ContainerStarted","Data":"267f204142b27d3958d7f4ebddaa9c82a2b9ed8a60f0b8fe49cf7e29e5774216"}
Jul 23 15:27:58.411821 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:27:58.411680    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7f6db9fe-bff779489-c55vd"
Jul 23 15:27:58.433023 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:27:58.432932    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7f6db9fe-bff779489-c55vd" podStartSLOduration=2.011286213 podStartE2EDuration="2.432912941s" totalImagesPullingTime="421.626728ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 15:27:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 15:27:58.432644648 +0000 UTC m=+2119.940518396" watchObservedRunningTime="2026-07-23 15:27:58.432912941 +0000 UTC m=+2119.940786689"
Jul 23 15:28:01.584262 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:28:01.584214    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cgwsg"]
Jul 23 15:28:04.421748 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:28:04.421721    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7f6db9fe-bff779489-c55vd"
Jul 23 15:28:18.415094 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:28:18.415058    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7f6db9fe-bff779489-c55vd"]
Jul 23 15:28:18.415625 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:28:18.415286    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7f6db9fe-bff779489-c55vd" podUID="d3691bff-328c-4586-8f3f-4cafd5a8f05c" containerName="maas-api" containerID="cri-o://32e8768b131b2b8ebf6805ab3e398e546d4e49af4d0822d3b01e02780c6dd049" gracePeriod=30
Jul 23 15:28:18.662085 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:28:18.662061    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7f6db9fe-bff779489-c55vd"
Jul 23 15:28:18.741019 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:28:18.740985    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3691bff-328c-4586-8f3f-4cafd5a8f05c-maas-api-tls\") pod \"d3691bff-328c-4586-8f3f-4cafd5a8f05c\" (UID: \"d3691bff-328c-4586-8f3f-4cafd5a8f05c\") "
Jul 23 15:28:18.741019 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:28:18.741024    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pqwfs\" (UniqueName: \"kubernetes.io/projected/d3691bff-328c-4586-8f3f-4cafd5a8f05c-kube-api-access-pqwfs\") pod \"d3691bff-328c-4586-8f3f-4cafd5a8f05c\" (UID: \"d3691bff-328c-4586-8f3f-4cafd5a8f05c\") "
Jul 23 15:28:18.741270 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:28:18.741045    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3691bff-328c-4586-8f3f-4cafd5a8f05c-openshift-service-ca\") pod \"d3691bff-328c-4586-8f3f-4cafd5a8f05c\" (UID: \"d3691bff-328c-4586-8f3f-4cafd5a8f05c\") "
Jul 23 15:28:18.741475 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:28:18.741450    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3691bff-328c-4586-8f3f-4cafd5a8f05c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d3691bff-328c-4586-8f3f-4cafd5a8f05c" (UID: "d3691bff-328c-4586-8f3f-4cafd5a8f05c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 15:28:18.743397 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:28:18.743373    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3691bff-328c-4586-8f3f-4cafd5a8f05c-kube-api-access-pqwfs" (OuterVolumeSpecName: "kube-api-access-pqwfs") pod "d3691bff-328c-4586-8f3f-4cafd5a8f05c" (UID: "d3691bff-328c-4586-8f3f-4cafd5a8f05c"). InnerVolumeSpecName "kube-api-access-pqwfs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 15:28:18.743397 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:28:18.743377    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3691bff-328c-4586-8f3f-4cafd5a8f05c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d3691bff-328c-4586-8f3f-4cafd5a8f05c" (UID: "d3691bff-328c-4586-8f3f-4cafd5a8f05c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 15:28:18.841952 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:28:18.841911    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3691bff-328c-4586-8f3f-4cafd5a8f05c-maas-api-tls\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Jul 23 15:28:18.841952 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:28:18.841943    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqwfs\" (UniqueName: \"kubernetes.io/projected/d3691bff-328c-4586-8f3f-4cafd5a8f05c-kube-api-access-pqwfs\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Jul 23 15:28:18.841952 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:28:18.841953    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3691bff-328c-4586-8f3f-4cafd5a8f05c-openshift-service-ca\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Jul 23 15:28:19.514416 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:28:19.514380    2560 generic.go:356] "Generic (PLEG): container finished" podID="d3691bff-328c-4586-8f3f-4cafd5a8f05c" containerID="32e8768b131b2b8ebf6805ab3e398e546d4e49af4d0822d3b01e02780c6dd049" exitCode=0
Jul 23 15:28:19.514832 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:28:19.514429    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7f6db9fe-bff779489-c55vd"
Jul 23 15:28:19.514832 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:28:19.514471    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7f6db9fe-bff779489-c55vd" event={"ID":"d3691bff-328c-4586-8f3f-4cafd5a8f05c","Type":"ContainerDied","Data":"32e8768b131b2b8ebf6805ab3e398e546d4e49af4d0822d3b01e02780c6dd049"}
Jul 23 15:28:19.514832 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:28:19.514528    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7f6db9fe-bff779489-c55vd" event={"ID":"d3691bff-328c-4586-8f3f-4cafd5a8f05c","Type":"ContainerDied","Data":"267f204142b27d3958d7f4ebddaa9c82a2b9ed8a60f0b8fe49cf7e29e5774216"}
Jul 23 15:28:19.514832 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:28:19.514551    2560 scope.go:117] "RemoveContainer" containerID="32e8768b131b2b8ebf6805ab3e398e546d4e49af4d0822d3b01e02780c6dd049"
Jul 23 15:28:19.524029 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:28:19.524007    2560 scope.go:117] "RemoveContainer" containerID="32e8768b131b2b8ebf6805ab3e398e546d4e49af4d0822d3b01e02780c6dd049"
Jul 23 15:28:19.524344 ip-10-0-141-105 kubenswrapper[2560]: E0723 15:28:19.524324    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32e8768b131b2b8ebf6805ab3e398e546d4e49af4d0822d3b01e02780c6dd049\": container with ID starting with 32e8768b131b2b8ebf6805ab3e398e546d4e49af4d0822d3b01e02780c6dd049 not found: ID does not exist" containerID="32e8768b131b2b8ebf6805ab3e398e546d4e49af4d0822d3b01e02780c6dd049"
Jul 23 15:28:19.524409 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:28:19.524354    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32e8768b131b2b8ebf6805ab3e398e546d4e49af4d0822d3b01e02780c6dd049"} err="failed to get container status \"32e8768b131b2b8ebf6805ab3e398e546d4e49af4d0822d3b01e02780c6dd049\": rpc error: code = NotFound desc = could not find container \"32e8768b131b2b8ebf6805ab3e398e546d4e49af4d0822d3b01e02780c6dd049\": container with ID starting with 32e8768b131b2b8ebf6805ab3e398e546d4e49af4d0822d3b01e02780c6dd049 not found: ID does not exist"
Jul 23 15:28:19.537983 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:28:19.537939    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7f6db9fe-bff779489-c55vd"]
Jul 23 15:28:19.539286 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:28:19.539261    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7f6db9fe-bff779489-c55vd"]
Jul 23 15:28:21.062553 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:28:21.062517    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d3691bff-328c-4586-8f3f-4cafd5a8f05c" path="/var/lib/kubelet/pods/d3691bff-328c-4586-8f3f-4cafd5a8f05c/volumes"
Jul 23 15:28:21.323565 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:28:21.323464    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cgwsg"]
Jul 23 15:28:52.521964 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:28:52.521931    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e2bd74f9-54b99c9bd5-bgpqk"]
Jul 23 15:28:52.522582 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:28:52.522514    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3691bff-328c-4586-8f3f-4cafd5a8f05c" containerName="maas-api"
Jul 23 15:28:52.522582 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:28:52.522536    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3691bff-328c-4586-8f3f-4cafd5a8f05c" containerName="maas-api"
Jul 23 15:28:52.522705 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:28:52.522622    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3691bff-328c-4586-8f3f-4cafd5a8f05c" containerName="maas-api"
Jul 23 15:28:52.525788 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:28:52.525763    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2bd74f9-54b99c9bd5-bgpqk"
Jul 23 15:28:52.528607 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:28:52.528577    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 23 15:28:52.528607 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:28:52.528597    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-e2bd74f9-serving-cert\""
Jul 23 15:28:52.528774 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:28:52.528602    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 23 15:28:52.529789 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:28:52.529768    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-rdm8l\""
Jul 23 15:28:52.535333 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:28:52.535304    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e2bd74f9-54b99c9bd5-bgpqk"]
Jul 23 15:28:52.637188 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:28:52.637148    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e5d8fe2d-433a-4dbd-9b38-4b589dd22cd6-maas-api-tls\") pod \"maas-api-e2e-mt-e2bd74f9-54b99c9bd5-bgpqk\" (UID: \"e5d8fe2d-433a-4dbd-9b38-4b589dd22cd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2bd74f9-54b99c9bd5-bgpqk"
Jul 23 15:28:52.637188 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:28:52.637190    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5d8fe2d-433a-4dbd-9b38-4b589dd22cd6-openshift-service-ca\") pod \"maas-api-e2e-mt-e2bd74f9-54b99c9bd5-bgpqk\" (UID: \"e5d8fe2d-433a-4dbd-9b38-4b589dd22cd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2bd74f9-54b99c9bd5-bgpqk"
Jul 23 15:28:52.637467 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:28:52.637287    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zmrw\" (UniqueName: \"kubernetes.io/projected/e5d8fe2d-433a-4dbd-9b38-4b589dd22cd6-kube-api-access-5zmrw\") pod \"maas-api-e2e-mt-e2bd74f9-54b99c9bd5-bgpqk\" (UID: \"e5d8fe2d-433a-4dbd-9b38-4b589dd22cd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2bd74f9-54b99c9bd5-bgpqk"
Jul 23 15:28:52.738676 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:28:52.738635    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zmrw\" (UniqueName: \"kubernetes.io/projected/e5d8fe2d-433a-4dbd-9b38-4b589dd22cd6-kube-api-access-5zmrw\") pod \"maas-api-e2e-mt-e2bd74f9-54b99c9bd5-bgpqk\" (UID: \"e5d8fe2d-433a-4dbd-9b38-4b589dd22cd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2bd74f9-54b99c9bd5-bgpqk"
Jul 23 15:28:52.738881 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:28:52.738739    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e5d8fe2d-433a-4dbd-9b38-4b589dd22cd6-maas-api-tls\") pod \"maas-api-e2e-mt-e2bd74f9-54b99c9bd5-bgpqk\" (UID: \"e5d8fe2d-433a-4dbd-9b38-4b589dd22cd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2bd74f9-54b99c9bd5-bgpqk"
Jul 23 15:28:52.738881 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:28:52.738772    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5d8fe2d-433a-4dbd-9b38-4b589dd22cd6-openshift-service-ca\") pod \"maas-api-e2e-mt-e2bd74f9-54b99c9bd5-bgpqk\" (UID: \"e5d8fe2d-433a-4dbd-9b38-4b589dd22cd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2bd74f9-54b99c9bd5-bgpqk"
Jul 23 15:28:52.739009 ip-10-0-141-105 kubenswrapper[2560]: E0723 15:28:52.738887    2560 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-e2bd74f9-serving-cert: secret "maas-api-e2e-mt-e2bd74f9-serving-cert" not found
Jul 23 15:28:52.739009 ip-10-0-141-105 kubenswrapper[2560]: E0723 15:28:52.738974    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5d8fe2d-433a-4dbd-9b38-4b589dd22cd6-maas-api-tls podName:e5d8fe2d-433a-4dbd-9b38-4b589dd22cd6 nodeName:}" failed. No retries permitted until 2026-07-23 15:28:53.238952675 +0000 UTC m=+2174.746826402 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e5d8fe2d-433a-4dbd-9b38-4b589dd22cd6-maas-api-tls") pod "maas-api-e2e-mt-e2bd74f9-54b99c9bd5-bgpqk" (UID: "e5d8fe2d-433a-4dbd-9b38-4b589dd22cd6") : secret "maas-api-e2e-mt-e2bd74f9-serving-cert" not found
Jul 23 15:28:52.739505 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:28:52.739481    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5d8fe2d-433a-4dbd-9b38-4b589dd22cd6-openshift-service-ca\") pod \"maas-api-e2e-mt-e2bd74f9-54b99c9bd5-bgpqk\" (UID: \"e5d8fe2d-433a-4dbd-9b38-4b589dd22cd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2bd74f9-54b99c9bd5-bgpqk"
Jul 23 15:28:52.749915 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:28:52.749884    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zmrw\" (UniqueName: \"kubernetes.io/projected/e5d8fe2d-433a-4dbd-9b38-4b589dd22cd6-kube-api-access-5zmrw\") pod \"maas-api-e2e-mt-e2bd74f9-54b99c9bd5-bgpqk\" (UID: \"e5d8fe2d-433a-4dbd-9b38-4b589dd22cd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2bd74f9-54b99c9bd5-bgpqk"
Jul 23 15:28:53.242912 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:28:53.242872    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e5d8fe2d-433a-4dbd-9b38-4b589dd22cd6-maas-api-tls\") pod \"maas-api-e2e-mt-e2bd74f9-54b99c9bd5-bgpqk\" (UID: \"e5d8fe2d-433a-4dbd-9b38-4b589dd22cd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2bd74f9-54b99c9bd5-bgpqk"
Jul 23 15:28:53.245487 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:28:53.245463    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e5d8fe2d-433a-4dbd-9b38-4b589dd22cd6-maas-api-tls\") pod \"maas-api-e2e-mt-e2bd74f9-54b99c9bd5-bgpqk\" (UID: \"e5d8fe2d-433a-4dbd-9b38-4b589dd22cd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2bd74f9-54b99c9bd5-bgpqk"
Jul 23 15:28:53.437707 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:28:53.437653    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2bd74f9-54b99c9bd5-bgpqk"
Jul 23 15:28:53.586052 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:28:53.586026    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e2bd74f9-54b99c9bd5-bgpqk"]
Jul 23 15:28:53.588825 ip-10-0-141-105 kubenswrapper[2560]: W0723 15:28:53.588797    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5d8fe2d_433a_4dbd_9b38_4b589dd22cd6.slice/crio-6aabb9037a119af836f6ee5d91d0266bdf405779174ddaaae68af7187366c01f WatchSource:0}: Error finding container 6aabb9037a119af836f6ee5d91d0266bdf405779174ddaaae68af7187366c01f: Status 404 returned error can't find the container with id 6aabb9037a119af836f6ee5d91d0266bdf405779174ddaaae68af7187366c01f
Jul 23 15:28:53.590196 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:28:53.590175    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 23 15:28:53.674538 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:28:53.674501    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2bd74f9-54b99c9bd5-bgpqk" event={"ID":"e5d8fe2d-433a-4dbd-9b38-4b589dd22cd6","Type":"ContainerStarted","Data":"6aabb9037a119af836f6ee5d91d0266bdf405779174ddaaae68af7187366c01f"}
Jul 23 15:28:54.680885 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:28:54.680851    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2bd74f9-54b99c9bd5-bgpqk" event={"ID":"e5d8fe2d-433a-4dbd-9b38-4b589dd22cd6","Type":"ContainerStarted","Data":"457a66d40a532af9ea919de33df57857cae5096a6ed0887a3e817b691a646ac7"}
Jul 23 15:28:54.681406 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:28:54.680978    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2bd74f9-54b99c9bd5-bgpqk"
Jul 23 15:28:54.697779 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:28:54.697724    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2bd74f9-54b99c9bd5-bgpqk" podStartSLOduration=2.345888813 podStartE2EDuration="2.697705542s" totalImagesPullingTime="351.816729ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 15:28:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 15:28:54.696449392 +0000 UTC m=+2176.204323163" watchObservedRunningTime="2026-07-23 15:28:54.697705542 +0000 UTC m=+2176.205579410"
Jul 23 15:28:58.619367 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:28:58.619314    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cgwsg"]
Jul 23 15:29:00.690889 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:29:00.690851    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2bd74f9-54b99c9bd5-bgpqk"
Jul 23 15:29:07.924604 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:29:07.924563    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-e2bd74f9/e2e-lifecycle-model-e2bd74f9-kserve-6d6f595d8c-qpnc9"]
Jul 23 15:29:07.928304 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:29:07.928282    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-e2bd74f9/e2e-lifecycle-model-e2bd74f9-kserve-6d6f595d8c-qpnc9"
Jul 23 15:29:07.932344 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:29:07.932322    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-e2bd74f9\"/\"e2e-lifecycle-model-e2bd74f9-kserve-self-signed-certs\""
Jul 23 15:29:07.932344 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:29:07.932334    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-e2bd74f9\"/\"kube-root-ca.crt\""
Jul 23 15:29:07.932519 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:29:07.932335    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-e2bd74f9\"/\"openshift-service-ca.crt\""
Jul 23 15:29:07.932519 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:29:07.932336    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-e2bd74f9\"/\"default-dockercfg-btjdh\""
Jul 23 15:29:07.936515 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:29:07.936488    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-e2bd74f9/e2e-lifecycle-model-e2bd74f9-kserve-6d6f595d8c-qpnc9"]
Jul 23 15:29:07.969191 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:29:07.969146    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba3bbca3-ef86-4ea7-a824-dffa5962808d-tmp-dir\") pod \"e2e-lifecycle-model-e2bd74f9-kserve-6d6f595d8c-qpnc9\" (UID: \"ba3bbca3-ef86-4ea7-a824-dffa5962808d\") " pod="ai-tenant-e2e-mt-e2bd74f9/e2e-lifecycle-model-e2bd74f9-kserve-6d6f595d8c-qpnc9"
Jul 23 15:29:07.969392 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:29:07.969224    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba3bbca3-ef86-4ea7-a824-dffa5962808d-home\") pod \"e2e-lifecycle-model-e2bd74f9-kserve-6d6f595d8c-qpnc9\" (UID: \"ba3bbca3-ef86-4ea7-a824-dffa5962808d\") " pod="ai-tenant-e2e-mt-e2bd74f9/e2e-lifecycle-model-e2bd74f9-kserve-6d6f595d8c-qpnc9"
Jul 23 15:29:07.969392 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:29:07.969289    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba3bbca3-ef86-4ea7-a824-dffa5962808d-tls-certs\") pod \"e2e-lifecycle-model-e2bd74f9-kserve-6d6f595d8c-qpnc9\" (UID: \"ba3bbca3-ef86-4ea7-a824-dffa5962808d\") " pod="ai-tenant-e2e-mt-e2bd74f9/e2e-lifecycle-model-e2bd74f9-kserve-6d6f595d8c-qpnc9"
Jul 23 15:29:07.969392 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:29:07.969341    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba3bbca3-ef86-4ea7-a824-dffa5962808d-dshm\") pod \"e2e-lifecycle-model-e2bd74f9-kserve-6d6f595d8c-qpnc9\" (UID: \"ba3bbca3-ef86-4ea7-a824-dffa5962808d\") " pod="ai-tenant-e2e-mt-e2bd74f9/e2e-lifecycle-model-e2bd74f9-kserve-6d6f595d8c-qpnc9"
Jul 23 15:29:07.969540 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:29:07.969426    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpkwd\" (UniqueName: \"kubernetes.io/projected/ba3bbca3-ef86-4ea7-a824-dffa5962808d-kube-api-access-rpkwd\") pod \"e2e-lifecycle-model-e2bd74f9-kserve-6d6f595d8c-qpnc9\" (UID: \"ba3bbca3-ef86-4ea7-a824-dffa5962808d\") " pod="ai-tenant-e2e-mt-e2bd74f9/e2e-lifecycle-model-e2bd74f9-kserve-6d6f595d8c-qpnc9"
Jul 23 15:29:07.969540 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:29:07.969461    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba3bbca3-ef86-4ea7-a824-dffa5962808d-model-cache\") pod \"e2e-lifecycle-model-e2bd74f9-kserve-6d6f595d8c-qpnc9\" (UID: \"ba3bbca3-ef86-4ea7-a824-dffa5962808d\") " pod="ai-tenant-e2e-mt-e2bd74f9/e2e-lifecycle-model-e2bd74f9-kserve-6d6f595d8c-qpnc9"
Jul 23 15:29:08.070622 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:29:08.070581    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpkwd\" (UniqueName: \"kubernetes.io/projected/ba3bbca3-ef86-4ea7-a824-dffa5962808d-kube-api-access-rpkwd\") pod \"e2e-lifecycle-model-e2bd74f9-kserve-6d6f595d8c-qpnc9\" (UID: \"ba3bbca3-ef86-4ea7-a824-dffa5962808d\") " pod="ai-tenant-e2e-mt-e2bd74f9/e2e-lifecycle-model-e2bd74f9-kserve-6d6f595d8c-qpnc9"
Jul 23 15:29:08.070816 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:29:08.070630    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba3bbca3-ef86-4ea7-a824-dffa5962808d-model-cache\") pod \"e2e-lifecycle-model-e2bd74f9-kserve-6d6f595d8c-qpnc9\" (UID: \"ba3bbca3-ef86-4ea7-a824-dffa5962808d\") " pod="ai-tenant-e2e-mt-e2bd74f9/e2e-lifecycle-model-e2bd74f9-kserve-6d6f595d8c-qpnc9"
Jul 23 15:29:08.070816 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:29:08.070701    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba3bbca3-ef86-4ea7-a824-dffa5962808d-tmp-dir\") pod \"e2e-lifecycle-model-e2bd74f9-kserve-6d6f595d8c-qpnc9\" (UID: \"ba3bbca3-ef86-4ea7-a824-dffa5962808d\") " pod="ai-tenant-e2e-mt-e2bd74f9/e2e-lifecycle-model-e2bd74f9-kserve-6d6f595d8c-qpnc9"
Jul 23 15:29:08.070816 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:29:08.070764    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba3bbca3-ef86-4ea7-a824-dffa5962808d-home\") pod \"e2e-lifecycle-model-e2bd74f9-kserve-6d6f595d8c-qpnc9\" (UID: \"ba3bbca3-ef86-4ea7-a824-dffa5962808d\") " pod="ai-tenant-e2e-mt-e2bd74f9/e2e-lifecycle-model-e2bd74f9-kserve-6d6f595d8c-qpnc9"
Jul 23 15:29:08.070816 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:29:08.070798    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba3bbca3-ef86-4ea7-a824-dffa5962808d-tls-certs\") pod \"e2e-lifecycle-model-e2bd74f9-kserve-6d6f595d8c-qpnc9\" (UID: \"ba3bbca3-ef86-4ea7-a824-dffa5962808d\") " pod="ai-tenant-e2e-mt-e2bd74f9/e2e-lifecycle-model-e2bd74f9-kserve-6d6f595d8c-qpnc9"
Jul 23 15:29:08.071069 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:29:08.070862    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba3bbca3-ef86-4ea7-a824-dffa5962808d-dshm\") pod \"e2e-lifecycle-model-e2bd74f9-kserve-6d6f595d8c-qpnc9\" (UID: \"ba3bbca3-ef86-4ea7-a824-dffa5962808d\") " pod="ai-tenant-e2e-mt-e2bd74f9/e2e-lifecycle-model-e2bd74f9-kserve-6d6f595d8c-qpnc9"
Jul 23 15:29:08.071117 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:29:08.071089    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba3bbca3-ef86-4ea7-a824-dffa5962808d-model-cache\") pod \"e2e-lifecycle-model-e2bd74f9-kserve-6d6f595d8c-qpnc9\" (UID: \"ba3bbca3-ef86-4ea7-a824-dffa5962808d\") " pod="ai-tenant-e2e-mt-e2bd74f9/e2e-lifecycle-model-e2bd74f9-kserve-6d6f595d8c-qpnc9"
Jul 23 15:29:08.071206 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:29:08.071186    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba3bbca3-ef86-4ea7-a824-dffa5962808d-tmp-dir\") pod \"e2e-lifecycle-model-e2bd74f9-kserve-6d6f595d8c-qpnc9\" (UID: \"ba3bbca3-ef86-4ea7-a824-dffa5962808d\") " pod="ai-tenant-e2e-mt-e2bd74f9/e2e-lifecycle-model-e2bd74f9-kserve-6d6f595d8c-qpnc9"
Jul 23 15:29:08.071304 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:29:08.071280    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba3bbca3-ef86-4ea7-a824-dffa5962808d-home\") pod \"e2e-lifecycle-model-e2bd74f9-kserve-6d6f595d8c-qpnc9\" (UID: \"ba3bbca3-ef86-4ea7-a824-dffa5962808d\") " pod="ai-tenant-e2e-mt-e2bd74f9/e2e-lifecycle-model-e2bd74f9-kserve-6d6f595d8c-qpnc9"
Jul 23 15:29:08.073601 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:29:08.073577    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba3bbca3-ef86-4ea7-a824-dffa5962808d-dshm\") pod \"e2e-lifecycle-model-e2bd74f9-kserve-6d6f595d8c-qpnc9\" (UID: \"ba3bbca3-ef86-4ea7-a824-dffa5962808d\") " pod="ai-tenant-e2e-mt-e2bd74f9/e2e-lifecycle-model-e2bd74f9-kserve-6d6f595d8c-qpnc9"
Jul 23 15:29:08.073698 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:29:08.073581    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba3bbca3-ef86-4ea7-a824-dffa5962808d-tls-certs\") pod \"e2e-lifecycle-model-e2bd74f9-kserve-6d6f595d8c-qpnc9\" (UID: \"ba3bbca3-ef86-4ea7-a824-dffa5962808d\") " pod="ai-tenant-e2e-mt-e2bd74f9/e2e-lifecycle-model-e2bd74f9-kserve-6d6f595d8c-qpnc9"
Jul 23 15:29:08.079496 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:29:08.079472    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpkwd\" (UniqueName: \"kubernetes.io/projected/ba3bbca3-ef86-4ea7-a824-dffa5962808d-kube-api-access-rpkwd\") pod \"e2e-lifecycle-model-e2bd74f9-kserve-6d6f595d8c-qpnc9\" (UID: \"ba3bbca3-ef86-4ea7-a824-dffa5962808d\") " pod="ai-tenant-e2e-mt-e2bd74f9/e2e-lifecycle-model-e2bd74f9-kserve-6d6f595d8c-qpnc9"
Jul 23 15:29:08.237849 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:29:08.237815    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-e2bd74f9/e2e-lifecycle-model-e2bd74f9-kserve-6d6f595d8c-qpnc9"
Jul 23 15:29:08.374215 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:29:08.374181    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-e2bd74f9/e2e-lifecycle-model-e2bd74f9-kserve-6d6f595d8c-qpnc9"]
Jul 23 15:29:08.377099 ip-10-0-141-105 kubenswrapper[2560]: W0723 15:29:08.377067    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podba3bbca3_ef86_4ea7_a824_dffa5962808d.slice/crio-afc47fb6464af6351b0f166700fa1fbb38b2e7e6cee29285230d01db2491c9b1 WatchSource:0}: Error finding container afc47fb6464af6351b0f166700fa1fbb38b2e7e6cee29285230d01db2491c9b1: Status 404 returned error can't find the container with id afc47fb6464af6351b0f166700fa1fbb38b2e7e6cee29285230d01db2491c9b1
Jul 23 15:29:08.747376 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:29:08.747339    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-e2bd74f9/e2e-lifecycle-model-e2bd74f9-kserve-6d6f595d8c-qpnc9" event={"ID":"ba3bbca3-ef86-4ea7-a824-dffa5962808d","Type":"ContainerStarted","Data":"9f8ed28f660f4b45cf72f8508c280b27e9f536bd1579f9fc6f429f8c53e64490"}
Jul 23 15:29:08.747376 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:29:08.747383    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-e2bd74f9/e2e-lifecycle-model-e2bd74f9-kserve-6d6f595d8c-qpnc9" event={"ID":"ba3bbca3-ef86-4ea7-a824-dffa5962808d","Type":"ContainerStarted","Data":"afc47fb6464af6351b0f166700fa1fbb38b2e7e6cee29285230d01db2491c9b1"}
Jul 23 15:29:08.765636 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:29:08.765585    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-e2bd74f9/e2e-lifecycle-model-e2bd74f9-kserve-6d6f595d8c-qpnc9" podStartSLOduration=1.7655706009999999 podStartE2EDuration="1.765570601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 15:29:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 15:29:08.764486505 +0000 UTC m=+2190.272360253" watchObservedRunningTime="2026-07-23 15:29:08.765570601 +0000 UTC m=+2190.273444396"
Jul 23 15:29:11.424299 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:29:11.424266    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cgwsg"]
Jul 23 15:29:18.238306 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:29:18.238262    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-e2bd74f9/e2e-lifecycle-model-e2bd74f9-kserve-6d6f595d8c-qpnc9"
Jul 23 15:29:18.238306 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:29:18.238314    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-e2bd74f9/e2e-lifecycle-model-e2bd74f9-kserve-6d6f595d8c-qpnc9"
Jul 23 15:29:18.250957 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:29:18.250919    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-e2bd74f9/e2e-lifecycle-model-e2bd74f9-kserve-6d6f595d8c-qpnc9"
Jul 23 15:29:18.808468 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:29:18.808439    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-e2bd74f9/e2e-lifecycle-model-e2bd74f9-kserve-6d6f595d8c-qpnc9"
Jul 23 15:29:27.120993 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:29:27.120959    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cgwsg"]
Jul 23 15:29:44.625523 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:29:44.625490    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cgwsg"]
Jul 23 15:29:46.219439 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:29:46.219402    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e2bd74f9-54b99c9bd5-bgpqk"]
Jul 23 15:29:46.219819 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:29:46.219626    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2bd74f9-54b99c9bd5-bgpqk" podUID="e5d8fe2d-433a-4dbd-9b38-4b589dd22cd6" containerName="maas-api" containerID="cri-o://457a66d40a532af9ea919de33df57857cae5096a6ed0887a3e817b691a646ac7" gracePeriod=30
Jul 23 15:29:46.463778 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:29:46.463751    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2bd74f9-54b99c9bd5-bgpqk"
Jul 23 15:29:46.625340 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:29:46.625223    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5d8fe2d-433a-4dbd-9b38-4b589dd22cd6-openshift-service-ca\") pod \"e5d8fe2d-433a-4dbd-9b38-4b589dd22cd6\" (UID: \"e5d8fe2d-433a-4dbd-9b38-4b589dd22cd6\") "
Jul 23 15:29:46.625340 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:29:46.625328    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5zmrw\" (UniqueName: \"kubernetes.io/projected/e5d8fe2d-433a-4dbd-9b38-4b589dd22cd6-kube-api-access-5zmrw\") pod \"e5d8fe2d-433a-4dbd-9b38-4b589dd22cd6\" (UID: \"e5d8fe2d-433a-4dbd-9b38-4b589dd22cd6\") "
Jul 23 15:29:46.625561 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:29:46.625373    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e5d8fe2d-433a-4dbd-9b38-4b589dd22cd6-maas-api-tls\") pod \"e5d8fe2d-433a-4dbd-9b38-4b589dd22cd6\" (UID: \"e5d8fe2d-433a-4dbd-9b38-4b589dd22cd6\") "
Jul 23 15:29:46.625640 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:29:46.625612    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5d8fe2d-433a-4dbd-9b38-4b589dd22cd6-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e5d8fe2d-433a-4dbd-9b38-4b589dd22cd6" (UID: "e5d8fe2d-433a-4dbd-9b38-4b589dd22cd6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 15:29:46.627524 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:29:46.627500    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5d8fe2d-433a-4dbd-9b38-4b589dd22cd6-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e5d8fe2d-433a-4dbd-9b38-4b589dd22cd6" (UID: "e5d8fe2d-433a-4dbd-9b38-4b589dd22cd6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 15:29:46.627524 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:29:46.627504    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5d8fe2d-433a-4dbd-9b38-4b589dd22cd6-kube-api-access-5zmrw" (OuterVolumeSpecName: "kube-api-access-5zmrw") pod "e5d8fe2d-433a-4dbd-9b38-4b589dd22cd6" (UID: "e5d8fe2d-433a-4dbd-9b38-4b589dd22cd6"). InnerVolumeSpecName "kube-api-access-5zmrw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 15:29:46.726999 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:29:46.726968    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zmrw\" (UniqueName: \"kubernetes.io/projected/e5d8fe2d-433a-4dbd-9b38-4b589dd22cd6-kube-api-access-5zmrw\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Jul 23 15:29:46.726999 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:29:46.726995    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e5d8fe2d-433a-4dbd-9b38-4b589dd22cd6-maas-api-tls\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Jul 23 15:29:46.726999 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:29:46.727004    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5d8fe2d-433a-4dbd-9b38-4b589dd22cd6-openshift-service-ca\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Jul 23 15:29:46.922862 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:29:46.922777    2560 generic.go:356] "Generic (PLEG): container finished" podID="e5d8fe2d-433a-4dbd-9b38-4b589dd22cd6" containerID="457a66d40a532af9ea919de33df57857cae5096a6ed0887a3e817b691a646ac7" exitCode=0
Jul 23 15:29:46.922862 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:29:46.922820    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2bd74f9-54b99c9bd5-bgpqk" event={"ID":"e5d8fe2d-433a-4dbd-9b38-4b589dd22cd6","Type":"ContainerDied","Data":"457a66d40a532af9ea919de33df57857cae5096a6ed0887a3e817b691a646ac7"}
Jul 23 15:29:46.923048 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:29:46.922875    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2bd74f9-54b99c9bd5-bgpqk" event={"ID":"e5d8fe2d-433a-4dbd-9b38-4b589dd22cd6","Type":"ContainerDied","Data":"6aabb9037a119af836f6ee5d91d0266bdf405779174ddaaae68af7187366c01f"}
Jul 23 15:29:46.923048 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:29:46.922900    2560 scope.go:117] "RemoveContainer" containerID="457a66d40a532af9ea919de33df57857cae5096a6ed0887a3e817b691a646ac7"
Jul 23 15:29:46.923048 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:29:46.922831    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2bd74f9-54b99c9bd5-bgpqk"
Jul 23 15:29:46.931979 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:29:46.931959    2560 scope.go:117] "RemoveContainer" containerID="457a66d40a532af9ea919de33df57857cae5096a6ed0887a3e817b691a646ac7"
Jul 23 15:29:46.932259 ip-10-0-141-105 kubenswrapper[2560]: E0723 15:29:46.932210    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"457a66d40a532af9ea919de33df57857cae5096a6ed0887a3e817b691a646ac7\": container with ID starting with 457a66d40a532af9ea919de33df57857cae5096a6ed0887a3e817b691a646ac7 not found: ID does not exist" containerID="457a66d40a532af9ea919de33df57857cae5096a6ed0887a3e817b691a646ac7"
Jul 23 15:29:46.932329 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:29:46.932257    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"457a66d40a532af9ea919de33df57857cae5096a6ed0887a3e817b691a646ac7"} err="failed to get container status \"457a66d40a532af9ea919de33df57857cae5096a6ed0887a3e817b691a646ac7\": rpc error: code = NotFound desc = could not find container \"457a66d40a532af9ea919de33df57857cae5096a6ed0887a3e817b691a646ac7\": container with ID starting with 457a66d40a532af9ea919de33df57857cae5096a6ed0887a3e817b691a646ac7 not found: ID does not exist"
Jul 23 15:29:46.945433 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:29:46.945413    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e2bd74f9-54b99c9bd5-bgpqk"]
Jul 23 15:29:46.949147 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:29:46.949125    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e2bd74f9-54b99c9bd5-bgpqk"]
Jul 23 15:29:47.063100 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:29:47.063066    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e5d8fe2d-433a-4dbd-9b38-4b589dd22cd6" path="/var/lib/kubelet/pods/e5d8fe2d-433a-4dbd-9b38-4b589dd22cd6/volumes"
Jul 23 15:29:53.447050 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:29:53.447011    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cgwsg"]
Jul 23 15:29:58.335988 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:29:58.335950    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-e2bd74f9/e2e-lifecycle-model-e2bd74f9-kserve-6d6f595d8c-qpnc9"]
Jul 23 15:30:13.356686 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:30:13.356640    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-e2bd74f9/e2e-lifecycle-model-e2bd74f9-kserve-6d6f595d8c-qpnc9" podUID="ba3bbca3-ef86-4ea7-a824-dffa5962808d" containerName="main" containerID="cri-o://9f8ed28f660f4b45cf72f8508c280b27e9f536bd1579f9fc6f429f8c53e64490" gracePeriod=45
Jul 23 15:30:13.594700 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:30:13.594676    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-e2bd74f9/e2e-lifecycle-model-e2bd74f9-kserve-6d6f595d8c-qpnc9"
Jul 23 15:30:13.662088 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:30:13.662011    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba3bbca3-ef86-4ea7-a824-dffa5962808d-model-cache\") pod \"ba3bbca3-ef86-4ea7-a824-dffa5962808d\" (UID: \"ba3bbca3-ef86-4ea7-a824-dffa5962808d\") "
Jul 23 15:30:13.662269 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:30:13.662114    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba3bbca3-ef86-4ea7-a824-dffa5962808d-tls-certs\") pod \"ba3bbca3-ef86-4ea7-a824-dffa5962808d\" (UID: \"ba3bbca3-ef86-4ea7-a824-dffa5962808d\") "
Jul 23 15:30:13.662269 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:30:13.662164    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba3bbca3-ef86-4ea7-a824-dffa5962808d-dshm\") pod \"ba3bbca3-ef86-4ea7-a824-dffa5962808d\" (UID: \"ba3bbca3-ef86-4ea7-a824-dffa5962808d\") "
Jul 23 15:30:13.662269 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:30:13.662191    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba3bbca3-ef86-4ea7-a824-dffa5962808d-tmp-dir\") pod \"ba3bbca3-ef86-4ea7-a824-dffa5962808d\" (UID: \"ba3bbca3-ef86-4ea7-a824-dffa5962808d\") "
Jul 23 15:30:13.662269 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:30:13.662261    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba3bbca3-ef86-4ea7-a824-dffa5962808d-home\") pod \"ba3bbca3-ef86-4ea7-a824-dffa5962808d\" (UID: \"ba3bbca3-ef86-4ea7-a824-dffa5962808d\") "
Jul 23 15:30:13.662491 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:30:13.662290    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba3bbca3-ef86-4ea7-a824-dffa5962808d-model-cache" (OuterVolumeSpecName: "model-cache") pod "ba3bbca3-ef86-4ea7-a824-dffa5962808d" (UID: "ba3bbca3-ef86-4ea7-a824-dffa5962808d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 15:30:13.662491 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:30:13.662303    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rpkwd\" (UniqueName: \"kubernetes.io/projected/ba3bbca3-ef86-4ea7-a824-dffa5962808d-kube-api-access-rpkwd\") pod \"ba3bbca3-ef86-4ea7-a824-dffa5962808d\" (UID: \"ba3bbca3-ef86-4ea7-a824-dffa5962808d\") "
Jul 23 15:30:13.662591 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:30:13.662558    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba3bbca3-ef86-4ea7-a824-dffa5962808d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ba3bbca3-ef86-4ea7-a824-dffa5962808d" (UID: "ba3bbca3-ef86-4ea7-a824-dffa5962808d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 15:30:13.662644 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:30:13.662608    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba3bbca3-ef86-4ea7-a824-dffa5962808d-model-cache\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Jul 23 15:30:13.662644 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:30:13.662613    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba3bbca3-ef86-4ea7-a824-dffa5962808d-home" (OuterVolumeSpecName: "home") pod "ba3bbca3-ef86-4ea7-a824-dffa5962808d" (UID: "ba3bbca3-ef86-4ea7-a824-dffa5962808d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 15:30:13.664513 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:30:13.664485    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba3bbca3-ef86-4ea7-a824-dffa5962808d-dshm" (OuterVolumeSpecName: "dshm") pod "ba3bbca3-ef86-4ea7-a824-dffa5962808d" (UID: "ba3bbca3-ef86-4ea7-a824-dffa5962808d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 15:30:13.664803 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:30:13.664771    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba3bbca3-ef86-4ea7-a824-dffa5962808d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ba3bbca3-ef86-4ea7-a824-dffa5962808d" (UID: "ba3bbca3-ef86-4ea7-a824-dffa5962808d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 15:30:13.664803 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:30:13.664776    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba3bbca3-ef86-4ea7-a824-dffa5962808d-kube-api-access-rpkwd" (OuterVolumeSpecName: "kube-api-access-rpkwd") pod "ba3bbca3-ef86-4ea7-a824-dffa5962808d" (UID: "ba3bbca3-ef86-4ea7-a824-dffa5962808d"). InnerVolumeSpecName "kube-api-access-rpkwd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 15:30:13.763351 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:30:13.763315    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba3bbca3-ef86-4ea7-a824-dffa5962808d-tls-certs\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Jul 23 15:30:13.763351 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:30:13.763344    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba3bbca3-ef86-4ea7-a824-dffa5962808d-dshm\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Jul 23 15:30:13.763351 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:30:13.763355    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba3bbca3-ef86-4ea7-a824-dffa5962808d-tmp-dir\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Jul 23 15:30:13.763550 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:30:13.763366    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba3bbca3-ef86-4ea7-a824-dffa5962808d-home\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Jul 23 15:30:13.763550 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:30:13.763377    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rpkwd\" (UniqueName: \"kubernetes.io/projected/ba3bbca3-ef86-4ea7-a824-dffa5962808d-kube-api-access-rpkwd\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Jul 23 15:30:14.048209 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:30:14.048170    2560 generic.go:356] "Generic (PLEG): container finished" podID="ba3bbca3-ef86-4ea7-a824-dffa5962808d" containerID="9f8ed28f660f4b45cf72f8508c280b27e9f536bd1579f9fc6f429f8c53e64490" exitCode=0
Jul 23 15:30:14.048412 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:30:14.048218    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-e2bd74f9/e2e-lifecycle-model-e2bd74f9-kserve-6d6f595d8c-qpnc9"
Jul 23 15:30:14.048412 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:30:14.048286    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-e2bd74f9/e2e-lifecycle-model-e2bd74f9-kserve-6d6f595d8c-qpnc9" event={"ID":"ba3bbca3-ef86-4ea7-a824-dffa5962808d","Type":"ContainerDied","Data":"9f8ed28f660f4b45cf72f8508c280b27e9f536bd1579f9fc6f429f8c53e64490"}
Jul 23 15:30:14.048412 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:30:14.048333    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-e2bd74f9/e2e-lifecycle-model-e2bd74f9-kserve-6d6f595d8c-qpnc9" event={"ID":"ba3bbca3-ef86-4ea7-a824-dffa5962808d","Type":"ContainerDied","Data":"afc47fb6464af6351b0f166700fa1fbb38b2e7e6cee29285230d01db2491c9b1"}
Jul 23 15:30:14.048412 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:30:14.048366    2560 scope.go:117] "RemoveContainer" containerID="9f8ed28f660f4b45cf72f8508c280b27e9f536bd1579f9fc6f429f8c53e64490"
Jul 23 15:30:14.057756 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:30:14.057735    2560 scope.go:117] "RemoveContainer" containerID="9f8ed28f660f4b45cf72f8508c280b27e9f536bd1579f9fc6f429f8c53e64490"
Jul 23 15:30:14.058028 ip-10-0-141-105 kubenswrapper[2560]: E0723 15:30:14.058009    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f8ed28f660f4b45cf72f8508c280b27e9f536bd1579f9fc6f429f8c53e64490\": container with ID starting with 9f8ed28f660f4b45cf72f8508c280b27e9f536bd1579f9fc6f429f8c53e64490 not found: ID does not exist" containerID="9f8ed28f660f4b45cf72f8508c280b27e9f536bd1579f9fc6f429f8c53e64490"
Jul 23 15:30:14.058114 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:30:14.058036    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f8ed28f660f4b45cf72f8508c280b27e9f536bd1579f9fc6f429f8c53e64490"} err="failed to get container status \"9f8ed28f660f4b45cf72f8508c280b27e9f536bd1579f9fc6f429f8c53e64490\": rpc error: code = NotFound desc = could not find container \"9f8ed28f660f4b45cf72f8508c280b27e9f536bd1579f9fc6f429f8c53e64490\": container with ID starting with 9f8ed28f660f4b45cf72f8508c280b27e9f536bd1579f9fc6f429f8c53e64490 not found: ID does not exist"
Jul 23 15:30:14.070523 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:30:14.070503    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-e2bd74f9/e2e-lifecycle-model-e2bd74f9-kserve-6d6f595d8c-qpnc9"]
Jul 23 15:30:14.074577 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:30:14.074557    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-e2bd74f9/e2e-lifecycle-model-e2bd74f9-kserve-6d6f595d8c-qpnc9"]
Jul 23 15:30:15.062422 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:30:15.062389    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ba3bbca3-ef86-4ea7-a824-dffa5962808d" path="/var/lib/kubelet/pods/ba3bbca3-ef86-4ea7-a824-dffa5962808d/volumes"
Jul 23 15:30:45.127812 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:30:45.127769    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cgwsg"]
Jul 23 15:30:54.726965 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:30:54.726931    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cgwsg"]
Jul 23 15:31:41.334705 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:31:41.334627    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ec2b36-7d9c4cfd7-7c7tm"]
Jul 23 15:31:41.335359 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:31:41.334990    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5d8fe2d-433a-4dbd-9b38-4b589dd22cd6" containerName="maas-api"
Jul 23 15:31:41.335359 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:31:41.335002    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5d8fe2d-433a-4dbd-9b38-4b589dd22cd6" containerName="maas-api"
Jul 23 15:31:41.335359 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:31:41.335024    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba3bbca3-ef86-4ea7-a824-dffa5962808d" containerName="main"
Jul 23 15:31:41.335359 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:31:41.335031    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba3bbca3-ef86-4ea7-a824-dffa5962808d" containerName="main"
Jul 23 15:31:41.335359 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:31:41.335092    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5d8fe2d-433a-4dbd-9b38-4b589dd22cd6" containerName="maas-api"
Jul 23 15:31:41.335359 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:31:41.335100    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba3bbca3-ef86-4ea7-a824-dffa5962808d" containerName="main"
Jul 23 15:31:41.338354 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:31:41.338336    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ec2b36-7d9c4cfd7-7c7tm"
Jul 23 15:31:41.341054 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:31:41.341033    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-ec2b36-serving-cert\""
Jul 23 15:31:41.341197 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:31:41.341167    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-rdm8l\""
Jul 23 15:31:41.341197 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:31:41.341180    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 23 15:31:41.342424 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:31:41.342390    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 23 15:31:41.347400 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:31:41.347377    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ec2b36-7d9c4cfd7-7c7tm"]
Jul 23 15:31:41.443913 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:31:41.443872    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6lfv\" (UniqueName: \"kubernetes.io/projected/19775886-99a4-4d0d-887c-32098c54fb83-kube-api-access-g6lfv\") pod \"maas-api-e2e-inf-a-ec2b36-7d9c4cfd7-7c7tm\" (UID: \"19775886-99a4-4d0d-887c-32098c54fb83\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ec2b36-7d9c4cfd7-7c7tm"
Jul 23 15:31:41.444108 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:31:41.443925    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/19775886-99a4-4d0d-887c-32098c54fb83-openshift-service-ca\") pod \"maas-api-e2e-inf-a-ec2b36-7d9c4cfd7-7c7tm\" (UID: \"19775886-99a4-4d0d-887c-32098c54fb83\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ec2b36-7d9c4cfd7-7c7tm"
Jul 23 15:31:41.444108 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:31:41.443951    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/19775886-99a4-4d0d-887c-32098c54fb83-maas-api-tls\") pod \"maas-api-e2e-inf-a-ec2b36-7d9c4cfd7-7c7tm\" (UID: \"19775886-99a4-4d0d-887c-32098c54fb83\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ec2b36-7d9c4cfd7-7c7tm"
Jul 23 15:31:41.545427 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:31:41.545385    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6lfv\" (UniqueName: \"kubernetes.io/projected/19775886-99a4-4d0d-887c-32098c54fb83-kube-api-access-g6lfv\") pod \"maas-api-e2e-inf-a-ec2b36-7d9c4cfd7-7c7tm\" (UID: \"19775886-99a4-4d0d-887c-32098c54fb83\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ec2b36-7d9c4cfd7-7c7tm"
Jul 23 15:31:41.545427 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:31:41.545428    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/19775886-99a4-4d0d-887c-32098c54fb83-openshift-service-ca\") pod \"maas-api-e2e-inf-a-ec2b36-7d9c4cfd7-7c7tm\" (UID: \"19775886-99a4-4d0d-887c-32098c54fb83\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ec2b36-7d9c4cfd7-7c7tm"
Jul 23 15:31:41.545693 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:31:41.545452    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/19775886-99a4-4d0d-887c-32098c54fb83-maas-api-tls\") pod \"maas-api-e2e-inf-a-ec2b36-7d9c4cfd7-7c7tm\" (UID: \"19775886-99a4-4d0d-887c-32098c54fb83\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ec2b36-7d9c4cfd7-7c7tm"
Jul 23 15:31:41.545693 ip-10-0-141-105 kubenswrapper[2560]: E0723 15:31:41.545588    2560 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-ec2b36-serving-cert: secret "maas-api-e2e-inf-a-ec2b36-serving-cert" not found
Jul 23 15:31:41.545693 ip-10-0-141-105 kubenswrapper[2560]: E0723 15:31:41.545638    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19775886-99a4-4d0d-887c-32098c54fb83-maas-api-tls podName:19775886-99a4-4d0d-887c-32098c54fb83 nodeName:}" failed. No retries permitted until 2026-07-23 15:31:42.045620972 +0000 UTC m=+2343.553494699 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/19775886-99a4-4d0d-887c-32098c54fb83-maas-api-tls") pod "maas-api-e2e-inf-a-ec2b36-7d9c4cfd7-7c7tm" (UID: "19775886-99a4-4d0d-887c-32098c54fb83") : secret "maas-api-e2e-inf-a-ec2b36-serving-cert" not found
Jul 23 15:31:41.546156 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:31:41.546130    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/19775886-99a4-4d0d-887c-32098c54fb83-openshift-service-ca\") pod \"maas-api-e2e-inf-a-ec2b36-7d9c4cfd7-7c7tm\" (UID: \"19775886-99a4-4d0d-887c-32098c54fb83\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ec2b36-7d9c4cfd7-7c7tm"
Jul 23 15:31:41.564792 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:31:41.564758    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6lfv\" (UniqueName: \"kubernetes.io/projected/19775886-99a4-4d0d-887c-32098c54fb83-kube-api-access-g6lfv\") pod \"maas-api-e2e-inf-a-ec2b36-7d9c4cfd7-7c7tm\" (UID: \"19775886-99a4-4d0d-887c-32098c54fb83\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ec2b36-7d9c4cfd7-7c7tm"
Jul 23 15:31:42.050320 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:31:42.050281    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/19775886-99a4-4d0d-887c-32098c54fb83-maas-api-tls\") pod \"maas-api-e2e-inf-a-ec2b36-7d9c4cfd7-7c7tm\" (UID: \"19775886-99a4-4d0d-887c-32098c54fb83\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ec2b36-7d9c4cfd7-7c7tm"
Jul 23 15:31:42.052951 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:31:42.052924    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/19775886-99a4-4d0d-887c-32098c54fb83-maas-api-tls\") pod \"maas-api-e2e-inf-a-ec2b36-7d9c4cfd7-7c7tm\" (UID: \"19775886-99a4-4d0d-887c-32098c54fb83\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ec2b36-7d9c4cfd7-7c7tm"
Jul 23 15:31:42.249656 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:31:42.249615    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ec2b36-7d9c4cfd7-7c7tm"
Jul 23 15:31:42.387741 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:31:42.387706    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ec2b36-7d9c4cfd7-7c7tm"]
Jul 23 15:31:42.390682 ip-10-0-141-105 kubenswrapper[2560]: W0723 15:31:42.390640    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19775886_99a4_4d0d_887c_32098c54fb83.slice/crio-c4b251113ac0cd7b33e10370d4a4445f8e15b96705478265ab66c226126d20db WatchSource:0}: Error finding container c4b251113ac0cd7b33e10370d4a4445f8e15b96705478265ab66c226126d20db: Status 404 returned error can't find the container with id c4b251113ac0cd7b33e10370d4a4445f8e15b96705478265ab66c226126d20db
Jul 23 15:31:42.450378 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:31:42.450344    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ec2b36-7d9c4cfd7-7c7tm" event={"ID":"19775886-99a4-4d0d-887c-32098c54fb83","Type":"ContainerStarted","Data":"c4b251113ac0cd7b33e10370d4a4445f8e15b96705478265ab66c226126d20db"}
Jul 23 15:31:43.456513 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:31:43.456474    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ec2b36-7d9c4cfd7-7c7tm" event={"ID":"19775886-99a4-4d0d-887c-32098c54fb83","Type":"ContainerStarted","Data":"d6a2a95969bf2a3d36ee3bef6fc77c9d23b4f174fe113d7bf455cd08b3ab8f54"}
Jul 23 15:31:43.456970 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:31:43.456548    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ec2b36-7d9c4cfd7-7c7tm"
Jul 23 15:31:43.474883 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:31:43.474836    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ec2b36-7d9c4cfd7-7c7tm" podStartSLOduration=2.050131563 podStartE2EDuration="2.474820788s" totalImagesPullingTime="424.689225ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 15:31:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 15:31:43.473479403 +0000 UTC m=+2344.981353151" watchObservedRunningTime="2026-07-23 15:31:43.474820788 +0000 UTC m=+2344.982694543"
Jul 23 15:31:46.731651 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:31:46.731614    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cgwsg"]
Jul 23 15:31:49.379177 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:31:49.379147    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5c302e-6b4ff7cf7f-x42cd"]
Jul 23 15:31:49.382956 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:31:49.382933    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5c302e-6b4ff7cf7f-x42cd"
Jul 23 15:31:49.386065 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:31:49.386041    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-5c302e-serving-cert\""
Jul 23 15:31:49.404000 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:31:49.403973    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5c302e-6b4ff7cf7f-x42cd"]
Jul 23 15:31:49.467768 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:31:49.467743    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ec2b36-7d9c4cfd7-7c7tm"
Jul 23 15:31:49.522189 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:31:49.522142    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjm8v\" (UniqueName: \"kubernetes.io/projected/cd8266d1-d547-4b02-bc6f-4972742ca2bb-kube-api-access-fjm8v\") pod \"maas-api-e2e-inf-b-5c302e-6b4ff7cf7f-x42cd\" (UID: \"cd8266d1-d547-4b02-bc6f-4972742ca2bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5c302e-6b4ff7cf7f-x42cd"
Jul 23 15:31:49.522369 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:31:49.522200    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd8266d1-d547-4b02-bc6f-4972742ca2bb-openshift-service-ca\") pod \"maas-api-e2e-inf-b-5c302e-6b4ff7cf7f-x42cd\" (UID: \"cd8266d1-d547-4b02-bc6f-4972742ca2bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5c302e-6b4ff7cf7f-x42cd"
Jul 23 15:31:49.522369 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:31:49.522281    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd8266d1-d547-4b02-bc6f-4972742ca2bb-maas-api-tls\") pod \"maas-api-e2e-inf-b-5c302e-6b4ff7cf7f-x42cd\" (UID: \"cd8266d1-d547-4b02-bc6f-4972742ca2bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5c302e-6b4ff7cf7f-x42cd"
Jul 23 15:31:49.623349 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:31:49.623315    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjm8v\" (UniqueName: \"kubernetes.io/projected/cd8266d1-d547-4b02-bc6f-4972742ca2bb-kube-api-access-fjm8v\") pod \"maas-api-e2e-inf-b-5c302e-6b4ff7cf7f-x42cd\" (UID: \"cd8266d1-d547-4b02-bc6f-4972742ca2bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5c302e-6b4ff7cf7f-x42cd"
Jul 23 15:31:49.623537 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:31:49.623353    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd8266d1-d547-4b02-bc6f-4972742ca2bb-openshift-service-ca\") pod \"maas-api-e2e-inf-b-5c302e-6b4ff7cf7f-x42cd\" (UID: \"cd8266d1-d547-4b02-bc6f-4972742ca2bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5c302e-6b4ff7cf7f-x42cd"
Jul 23 15:31:49.623537 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:31:49.623384    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd8266d1-d547-4b02-bc6f-4972742ca2bb-maas-api-tls\") pod \"maas-api-e2e-inf-b-5c302e-6b4ff7cf7f-x42cd\" (UID: \"cd8266d1-d547-4b02-bc6f-4972742ca2bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5c302e-6b4ff7cf7f-x42cd"
Jul 23 15:31:49.624054 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:31:49.624024    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd8266d1-d547-4b02-bc6f-4972742ca2bb-openshift-service-ca\") pod \"maas-api-e2e-inf-b-5c302e-6b4ff7cf7f-x42cd\" (UID: \"cd8266d1-d547-4b02-bc6f-4972742ca2bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5c302e-6b4ff7cf7f-x42cd"
Jul 23 15:31:49.625873 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:31:49.625854    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd8266d1-d547-4b02-bc6f-4972742ca2bb-maas-api-tls\") pod \"maas-api-e2e-inf-b-5c302e-6b4ff7cf7f-x42cd\" (UID: \"cd8266d1-d547-4b02-bc6f-4972742ca2bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5c302e-6b4ff7cf7f-x42cd"
Jul 23 15:31:49.646396 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:31:49.646345    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjm8v\" (UniqueName: \"kubernetes.io/projected/cd8266d1-d547-4b02-bc6f-4972742ca2bb-kube-api-access-fjm8v\") pod \"maas-api-e2e-inf-b-5c302e-6b4ff7cf7f-x42cd\" (UID: \"cd8266d1-d547-4b02-bc6f-4972742ca2bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5c302e-6b4ff7cf7f-x42cd"
Jul 23 15:31:49.704399 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:31:49.704368    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5c302e-6b4ff7cf7f-x42cd"
Jul 23 15:31:49.837090 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:31:49.837064    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5c302e-6b4ff7cf7f-x42cd"]
Jul 23 15:31:49.840423 ip-10-0-141-105 kubenswrapper[2560]: W0723 15:31:49.840393    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd8266d1_d547_4b02_bc6f_4972742ca2bb.slice/crio-105013889ebdba6b0bc1af901cc1ebf9cf8fbac25dcd37abf0b198e801f75e9d WatchSource:0}: Error finding container 105013889ebdba6b0bc1af901cc1ebf9cf8fbac25dcd37abf0b198e801f75e9d: Status 404 returned error can't find the container with id 105013889ebdba6b0bc1af901cc1ebf9cf8fbac25dcd37abf0b198e801f75e9d
Jul 23 15:31:50.491414 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:31:50.491322    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5c302e-6b4ff7cf7f-x42cd" event={"ID":"cd8266d1-d547-4b02-bc6f-4972742ca2bb","Type":"ContainerStarted","Data":"e3dc89cae5ecc1f1bf3fc80d6c879c5058bbac108708d1b6520595abf897fdde"}
Jul 23 15:31:50.491414 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:31:50.491377    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5c302e-6b4ff7cf7f-x42cd" event={"ID":"cd8266d1-d547-4b02-bc6f-4972742ca2bb","Type":"ContainerStarted","Data":"105013889ebdba6b0bc1af901cc1ebf9cf8fbac25dcd37abf0b198e801f75e9d"}
Jul 23 15:31:50.491414 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:31:50.491401    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5c302e-6b4ff7cf7f-x42cd"
Jul 23 15:31:50.510834 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:31:50.510783    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5c302e-6b4ff7cf7f-x42cd" podStartSLOduration=1.123043193 podStartE2EDuration="1.510769761s" totalImagesPullingTime="387.726568ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 15:31:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 15:31:50.508811048 +0000 UTC m=+2352.016684794" watchObservedRunningTime="2026-07-23 15:31:50.510769761 +0000 UTC m=+2352.018643508"
Jul 23 15:31:55.718725 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:31:55.718689    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cgwsg"]
Jul 23 15:31:56.502472 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:31:56.502443    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5c302e-6b4ff7cf7f-x42cd"
Jul 23 15:32:03.521369 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:32:03.521328    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-ec2b36/test-model-ec2b36-kserve-78f9cdb4c-7pklb"]
Jul 23 15:32:03.524287 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:32:03.524266    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ec2b36/test-model-ec2b36-kserve-78f9cdb4c-7pklb"
Jul 23 15:32:03.528691 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:32:03.528669    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-ec2b36\"/\"test-model-ec2b36-kserve-self-signed-certs\""
Jul 23 15:32:03.528691 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:32:03.528679    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-ec2b36\"/\"default-dockercfg-w5lbq\""
Jul 23 15:32:03.528905 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:32:03.528683    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-ec2b36\"/\"openshift-service-ca.crt\""
Jul 23 15:32:03.528905 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:32:03.528710    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-ec2b36\"/\"kube-root-ca.crt\""
Jul 23 15:32:03.533964 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:32:03.533945    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-ec2b36/test-model-ec2b36-kserve-78f9cdb4c-7pklb"]
Jul 23 15:32:03.551448 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:32:03.551423    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa906dd9-9cfe-4508-9494-a07531b06d91-home\") pod \"test-model-ec2b36-kserve-78f9cdb4c-7pklb\" (UID: \"aa906dd9-9cfe-4508-9494-a07531b06d91\") " pod="ai-tenant-e2e-inf-a-ec2b36/test-model-ec2b36-kserve-78f9cdb4c-7pklb"
Jul 23 15:32:03.551583 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:32:03.551473    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa906dd9-9cfe-4508-9494-a07531b06d91-tmp-dir\") pod \"test-model-ec2b36-kserve-78f9cdb4c-7pklb\" (UID: \"aa906dd9-9cfe-4508-9494-a07531b06d91\") " pod="ai-tenant-e2e-inf-a-ec2b36/test-model-ec2b36-kserve-78f9cdb4c-7pklb"
Jul 23 15:32:03.551637 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:32:03.551579    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa906dd9-9cfe-4508-9494-a07531b06d91-dshm\") pod \"test-model-ec2b36-kserve-78f9cdb4c-7pklb\" (UID: \"aa906dd9-9cfe-4508-9494-a07531b06d91\") " pod="ai-tenant-e2e-inf-a-ec2b36/test-model-ec2b36-kserve-78f9cdb4c-7pklb"
Jul 23 15:32:03.551678 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:32:03.551656    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa906dd9-9cfe-4508-9494-a07531b06d91-model-cache\") pod \"test-model-ec2b36-kserve-78f9cdb4c-7pklb\" (UID: \"aa906dd9-9cfe-4508-9494-a07531b06d91\") " pod="ai-tenant-e2e-inf-a-ec2b36/test-model-ec2b36-kserve-78f9cdb4c-7pklb"
Jul 23 15:32:03.551712 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:32:03.551683    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa906dd9-9cfe-4508-9494-a07531b06d91-tls-certs\") pod \"test-model-ec2b36-kserve-78f9cdb4c-7pklb\" (UID: \"aa906dd9-9cfe-4508-9494-a07531b06d91\") " pod="ai-tenant-e2e-inf-a-ec2b36/test-model-ec2b36-kserve-78f9cdb4c-7pklb"
Jul 23 15:32:03.551764 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:32:03.551713    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cknxv\" (UniqueName: \"kubernetes.io/projected/aa906dd9-9cfe-4508-9494-a07531b06d91-kube-api-access-cknxv\") pod \"test-model-ec2b36-kserve-78f9cdb4c-7pklb\" (UID: \"aa906dd9-9cfe-4508-9494-a07531b06d91\") " pod="ai-tenant-e2e-inf-a-ec2b36/test-model-ec2b36-kserve-78f9cdb4c-7pklb"
Jul 23 15:32:03.652284 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:32:03.652216    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa906dd9-9cfe-4508-9494-a07531b06d91-home\") pod \"test-model-ec2b36-kserve-78f9cdb4c-7pklb\" (UID: \"aa906dd9-9cfe-4508-9494-a07531b06d91\") " pod="ai-tenant-e2e-inf-a-ec2b36/test-model-ec2b36-kserve-78f9cdb4c-7pklb"
Jul 23 15:32:03.652484 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:32:03.652306    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa906dd9-9cfe-4508-9494-a07531b06d91-tmp-dir\") pod \"test-model-ec2b36-kserve-78f9cdb4c-7pklb\" (UID: \"aa906dd9-9cfe-4508-9494-a07531b06d91\") " pod="ai-tenant-e2e-inf-a-ec2b36/test-model-ec2b36-kserve-78f9cdb4c-7pklb"
Jul 23 15:32:03.652484 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:32:03.652361    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa906dd9-9cfe-4508-9494-a07531b06d91-dshm\") pod \"test-model-ec2b36-kserve-78f9cdb4c-7pklb\" (UID: \"aa906dd9-9cfe-4508-9494-a07531b06d91\") " pod="ai-tenant-e2e-inf-a-ec2b36/test-model-ec2b36-kserve-78f9cdb4c-7pklb"
Jul 23 15:32:03.652484 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:32:03.652418    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa906dd9-9cfe-4508-9494-a07531b06d91-model-cache\") pod \"test-model-ec2b36-kserve-78f9cdb4c-7pklb\" (UID: \"aa906dd9-9cfe-4508-9494-a07531b06d91\") " pod="ai-tenant-e2e-inf-a-ec2b36/test-model-ec2b36-kserve-78f9cdb4c-7pklb"
Jul 23 15:32:03.652484 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:32:03.652444    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa906dd9-9cfe-4508-9494-a07531b06d91-tls-certs\") pod \"test-model-ec2b36-kserve-78f9cdb4c-7pklb\" (UID: \"aa906dd9-9cfe-4508-9494-a07531b06d91\") " pod="ai-tenant-e2e-inf-a-ec2b36/test-model-ec2b36-kserve-78f9cdb4c-7pklb"
Jul 23 15:32:03.652695 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:32:03.652472    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cknxv\" (UniqueName: \"kubernetes.io/projected/aa906dd9-9cfe-4508-9494-a07531b06d91-kube-api-access-cknxv\") pod \"test-model-ec2b36-kserve-78f9cdb4c-7pklb\" (UID: \"aa906dd9-9cfe-4508-9494-a07531b06d91\") " pod="ai-tenant-e2e-inf-a-ec2b36/test-model-ec2b36-kserve-78f9cdb4c-7pklb"
Jul 23 15:32:03.652695 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:32:03.652677    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa906dd9-9cfe-4508-9494-a07531b06d91-home\") pod \"test-model-ec2b36-kserve-78f9cdb4c-7pklb\" (UID: \"aa906dd9-9cfe-4508-9494-a07531b06d91\") " pod="ai-tenant-e2e-inf-a-ec2b36/test-model-ec2b36-kserve-78f9cdb4c-7pklb"
Jul 23 15:32:03.652788 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:32:03.652715    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa906dd9-9cfe-4508-9494-a07531b06d91-tmp-dir\") pod \"test-model-ec2b36-kserve-78f9cdb4c-7pklb\" (UID: \"aa906dd9-9cfe-4508-9494-a07531b06d91\") " pod="ai-tenant-e2e-inf-a-ec2b36/test-model-ec2b36-kserve-78f9cdb4c-7pklb"
Jul 23 15:32:03.652788 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:32:03.652758    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa906dd9-9cfe-4508-9494-a07531b06d91-model-cache\") pod \"test-model-ec2b36-kserve-78f9cdb4c-7pklb\" (UID: \"aa906dd9-9cfe-4508-9494-a07531b06d91\") " pod="ai-tenant-e2e-inf-a-ec2b36/test-model-ec2b36-kserve-78f9cdb4c-7pklb"
Jul 23 15:32:03.654658 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:32:03.654638    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa906dd9-9cfe-4508-9494-a07531b06d91-dshm\") pod \"test-model-ec2b36-kserve-78f9cdb4c-7pklb\" (UID: \"aa906dd9-9cfe-4508-9494-a07531b06d91\") " pod="ai-tenant-e2e-inf-a-ec2b36/test-model-ec2b36-kserve-78f9cdb4c-7pklb"
Jul 23 15:32:03.654979 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:32:03.654963    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa906dd9-9cfe-4508-9494-a07531b06d91-tls-certs\") pod \"test-model-ec2b36-kserve-78f9cdb4c-7pklb\" (UID: \"aa906dd9-9cfe-4508-9494-a07531b06d91\") " pod="ai-tenant-e2e-inf-a-ec2b36/test-model-ec2b36-kserve-78f9cdb4c-7pklb"
Jul 23 15:32:03.660920 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:32:03.660899    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cknxv\" (UniqueName: \"kubernetes.io/projected/aa906dd9-9cfe-4508-9494-a07531b06d91-kube-api-access-cknxv\") pod \"test-model-ec2b36-kserve-78f9cdb4c-7pklb\" (UID: \"aa906dd9-9cfe-4508-9494-a07531b06d91\") " pod="ai-tenant-e2e-inf-a-ec2b36/test-model-ec2b36-kserve-78f9cdb4c-7pklb"
Jul 23 15:32:03.834213 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:32:03.834109    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ec2b36/test-model-ec2b36-kserve-78f9cdb4c-7pklb"
Jul 23 15:32:03.969504 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:32:03.969460    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-ec2b36/test-model-ec2b36-kserve-78f9cdb4c-7pklb"]
Jul 23 15:32:03.971929 ip-10-0-141-105 kubenswrapper[2560]: W0723 15:32:03.971895    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaa906dd9_9cfe_4508_9494_a07531b06d91.slice/crio-8ac878011f7f647ba51d2283b98dd6fefc9b20a6541c8b3136e3fbdbccddc0fb WatchSource:0}: Error finding container 8ac878011f7f647ba51d2283b98dd6fefc9b20a6541c8b3136e3fbdbccddc0fb: Status 404 returned error can't find the container with id 8ac878011f7f647ba51d2283b98dd6fefc9b20a6541c8b3136e3fbdbccddc0fb
Jul 23 15:32:04.558565 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:32:04.558520    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ec2b36/test-model-ec2b36-kserve-78f9cdb4c-7pklb" event={"ID":"aa906dd9-9cfe-4508-9494-a07531b06d91","Type":"ContainerStarted","Data":"63c1e09462e547be38e47d18cae82258d3034dc380c3eb905fb16133ef74d30c"}
Jul 23 15:32:04.558928 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:32:04.558575    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ec2b36/test-model-ec2b36-kserve-78f9cdb4c-7pklb" event={"ID":"aa906dd9-9cfe-4508-9494-a07531b06d91","Type":"ContainerStarted","Data":"8ac878011f7f647ba51d2283b98dd6fefc9b20a6541c8b3136e3fbdbccddc0fb"}
Jul 23 15:32:04.577603 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:32:04.577550    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-ec2b36/test-model-ec2b36-kserve-78f9cdb4c-7pklb" podStartSLOduration=1.577534298 podStartE2EDuration="1.577534298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 15:32:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 15:32:04.57658249 +0000 UTC m=+2366.084456297" watchObservedRunningTime="2026-07-23 15:32:04.577534298 +0000 UTC m=+2366.085408045"
Jul 23 15:32:08.631989 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:32:08.631949    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cgwsg"]
Jul 23 15:32:13.834495 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:32:13.834450    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-ec2b36/test-model-ec2b36-kserve-78f9cdb4c-7pklb"
Jul 23 15:32:13.835123 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:32:13.834724    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-ec2b36/test-model-ec2b36-kserve-78f9cdb4c-7pklb"
Jul 23 15:32:13.847372 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:32:13.847347    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-ec2b36/test-model-ec2b36-kserve-78f9cdb4c-7pklb"
Jul 23 15:32:14.617209 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:32:14.617177    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-ec2b36/test-model-ec2b36-kserve-78f9cdb4c-7pklb"
Jul 23 15:32:22.122181 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:32:22.122145    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-5c302e/test-model-5c302e-kserve-7d69b9749c-79gr4"]
Jul 23 15:32:22.126219 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:32:22.126194    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-5c302e/test-model-5c302e-kserve-7d69b9749c-79gr4"
Jul 23 15:32:22.129203 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:32:22.129184    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-5c302e\"/\"kube-root-ca.crt\""
Jul 23 15:32:22.130606 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:32:22.130582    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-5c302e\"/\"test-model-5c302e-kserve-self-signed-certs\""
Jul 23 15:32:22.130709 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:32:22.130589    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-5c302e\"/\"openshift-service-ca.crt\""
Jul 23 15:32:22.130709 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:32:22.130589    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-5c302e\"/\"default-dockercfg-cr7x9\""
Jul 23 15:32:22.135965 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:32:22.135935    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-5c302e/test-model-5c302e-kserve-7d69b9749c-79gr4"]
Jul 23 15:32:22.232224 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:32:22.232190    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8bb30844-3633-4f0f-9e0f-fe5fc798dd6d-model-cache\") pod \"test-model-5c302e-kserve-7d69b9749c-79gr4\" (UID: \"8bb30844-3633-4f0f-9e0f-fe5fc798dd6d\") " pod="ai-tenant-e2e-inf-b-5c302e/test-model-5c302e-kserve-7d69b9749c-79gr4"
Jul 23 15:32:22.232224 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:32:22.232225    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8bb30844-3633-4f0f-9e0f-fe5fc798dd6d-tmp-dir\") pod \"test-model-5c302e-kserve-7d69b9749c-79gr4\" (UID: \"8bb30844-3633-4f0f-9e0f-fe5fc798dd6d\") " pod="ai-tenant-e2e-inf-b-5c302e/test-model-5c302e-kserve-7d69b9749c-79gr4"
Jul 23 15:32:22.232510 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:32:22.232325    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8bb30844-3633-4f0f-9e0f-fe5fc798dd6d-home\") pod \"test-model-5c302e-kserve-7d69b9749c-79gr4\" (UID: \"8bb30844-3633-4f0f-9e0f-fe5fc798dd6d\") " pod="ai-tenant-e2e-inf-b-5c302e/test-model-5c302e-kserve-7d69b9749c-79gr4"
Jul 23 15:32:22.232510 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:32:22.232344    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8bb30844-3633-4f0f-9e0f-fe5fc798dd6d-tls-certs\") pod \"test-model-5c302e-kserve-7d69b9749c-79gr4\" (UID: \"8bb30844-3633-4f0f-9e0f-fe5fc798dd6d\") " pod="ai-tenant-e2e-inf-b-5c302e/test-model-5c302e-kserve-7d69b9749c-79gr4"
Jul 23 15:32:22.232510 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:32:22.232457    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5m6k5\" (UniqueName: \"kubernetes.io/projected/8bb30844-3633-4f0f-9e0f-fe5fc798dd6d-kube-api-access-5m6k5\") pod \"test-model-5c302e-kserve-7d69b9749c-79gr4\" (UID: \"8bb30844-3633-4f0f-9e0f-fe5fc798dd6d\") " pod="ai-tenant-e2e-inf-b-5c302e/test-model-5c302e-kserve-7d69b9749c-79gr4"
Jul 23 15:32:22.232510 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:32:22.232505    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8bb30844-3633-4f0f-9e0f-fe5fc798dd6d-dshm\") pod \"test-model-5c302e-kserve-7d69b9749c-79gr4\" (UID: \"8bb30844-3633-4f0f-9e0f-fe5fc798dd6d\") " pod="ai-tenant-e2e-inf-b-5c302e/test-model-5c302e-kserve-7d69b9749c-79gr4"
Jul 23 15:32:22.333152 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:32:22.333113    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5m6k5\" (UniqueName: \"kubernetes.io/projected/8bb30844-3633-4f0f-9e0f-fe5fc798dd6d-kube-api-access-5m6k5\") pod \"test-model-5c302e-kserve-7d69b9749c-79gr4\" (UID: \"8bb30844-3633-4f0f-9e0f-fe5fc798dd6d\") " pod="ai-tenant-e2e-inf-b-5c302e/test-model-5c302e-kserve-7d69b9749c-79gr4"
Jul 23 15:32:22.333152 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:32:22.333155    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8bb30844-3633-4f0f-9e0f-fe5fc798dd6d-dshm\") pod \"test-model-5c302e-kserve-7d69b9749c-79gr4\" (UID: \"8bb30844-3633-4f0f-9e0f-fe5fc798dd6d\") " pod="ai-tenant-e2e-inf-b-5c302e/test-model-5c302e-kserve-7d69b9749c-79gr4"
Jul 23 15:32:22.333423 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:32:22.333187    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8bb30844-3633-4f0f-9e0f-fe5fc798dd6d-model-cache\") pod \"test-model-5c302e-kserve-7d69b9749c-79gr4\" (UID: \"8bb30844-3633-4f0f-9e0f-fe5fc798dd6d\") " pod="ai-tenant-e2e-inf-b-5c302e/test-model-5c302e-kserve-7d69b9749c-79gr4"
Jul 23 15:32:22.333423 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:32:22.333201    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8bb30844-3633-4f0f-9e0f-fe5fc798dd6d-tmp-dir\") pod \"test-model-5c302e-kserve-7d69b9749c-79gr4\" (UID: \"8bb30844-3633-4f0f-9e0f-fe5fc798dd6d\") " pod="ai-tenant-e2e-inf-b-5c302e/test-model-5c302e-kserve-7d69b9749c-79gr4"
Jul 23 15:32:22.333423 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:32:22.333273    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8bb30844-3633-4f0f-9e0f-fe5fc798dd6d-home\") pod \"test-model-5c302e-kserve-7d69b9749c-79gr4\" (UID: \"8bb30844-3633-4f0f-9e0f-fe5fc798dd6d\") " pod="ai-tenant-e2e-inf-b-5c302e/test-model-5c302e-kserve-7d69b9749c-79gr4"
Jul 23 15:32:22.333423 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:32:22.333290    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8bb30844-3633-4f0f-9e0f-fe5fc798dd6d-tls-certs\") pod \"test-model-5c302e-kserve-7d69b9749c-79gr4\" (UID: \"8bb30844-3633-4f0f-9e0f-fe5fc798dd6d\") " pod="ai-tenant-e2e-inf-b-5c302e/test-model-5c302e-kserve-7d69b9749c-79gr4"
Jul 23 15:32:22.333719 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:32:22.333688    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8bb30844-3633-4f0f-9e0f-fe5fc798dd6d-tmp-dir\") pod \"test-model-5c302e-kserve-7d69b9749c-79gr4\" (UID: \"8bb30844-3633-4f0f-9e0f-fe5fc798dd6d\") " pod="ai-tenant-e2e-inf-b-5c302e/test-model-5c302e-kserve-7d69b9749c-79gr4"
Jul 23 15:32:22.333853 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:32:22.333698    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8bb30844-3633-4f0f-9e0f-fe5fc798dd6d-model-cache\") pod \"test-model-5c302e-kserve-7d69b9749c-79gr4\" (UID: \"8bb30844-3633-4f0f-9e0f-fe5fc798dd6d\") " pod="ai-tenant-e2e-inf-b-5c302e/test-model-5c302e-kserve-7d69b9749c-79gr4"
Jul 23 15:32:22.333853 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:32:22.333784    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8bb30844-3633-4f0f-9e0f-fe5fc798dd6d-home\") pod \"test-model-5c302e-kserve-7d69b9749c-79gr4\" (UID: \"8bb30844-3633-4f0f-9e0f-fe5fc798dd6d\") " pod="ai-tenant-e2e-inf-b-5c302e/test-model-5c302e-kserve-7d69b9749c-79gr4"
Jul 23 15:32:22.335594 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:32:22.335565    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8bb30844-3633-4f0f-9e0f-fe5fc798dd6d-dshm\") pod \"test-model-5c302e-kserve-7d69b9749c-79gr4\" (UID: \"8bb30844-3633-4f0f-9e0f-fe5fc798dd6d\") " pod="ai-tenant-e2e-inf-b-5c302e/test-model-5c302e-kserve-7d69b9749c-79gr4"
Jul 23 15:32:22.335755 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:32:22.335737    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8bb30844-3633-4f0f-9e0f-fe5fc798dd6d-tls-certs\") pod \"test-model-5c302e-kserve-7d69b9749c-79gr4\" (UID: \"8bb30844-3633-4f0f-9e0f-fe5fc798dd6d\") " pod="ai-tenant-e2e-inf-b-5c302e/test-model-5c302e-kserve-7d69b9749c-79gr4"
Jul 23 15:32:22.341711 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:32:22.341689    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5m6k5\" (UniqueName: \"kubernetes.io/projected/8bb30844-3633-4f0f-9e0f-fe5fc798dd6d-kube-api-access-5m6k5\") pod \"test-model-5c302e-kserve-7d69b9749c-79gr4\" (UID: \"8bb30844-3633-4f0f-9e0f-fe5fc798dd6d\") " pod="ai-tenant-e2e-inf-b-5c302e/test-model-5c302e-kserve-7d69b9749c-79gr4"
Jul 23 15:32:22.438739 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:32:22.438652    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-5c302e/test-model-5c302e-kserve-7d69b9749c-79gr4"
Jul 23 15:32:22.574643 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:32:22.574613    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-5c302e/test-model-5c302e-kserve-7d69b9749c-79gr4"]
Jul 23 15:32:22.577601 ip-10-0-141-105 kubenswrapper[2560]: W0723 15:32:22.577569    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8bb30844_3633_4f0f_9e0f_fe5fc798dd6d.slice/crio-dc64ef28d4d5791b23b5da5560b3fcff073c8568ea67a6b1effd6506b5158113 WatchSource:0}: Error finding container dc64ef28d4d5791b23b5da5560b3fcff073c8568ea67a6b1effd6506b5158113: Status 404 returned error can't find the container with id dc64ef28d4d5791b23b5da5560b3fcff073c8568ea67a6b1effd6506b5158113
Jul 23 15:32:22.646712 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:32:22.646682    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-5c302e/test-model-5c302e-kserve-7d69b9749c-79gr4" event={"ID":"8bb30844-3633-4f0f-9e0f-fe5fc798dd6d","Type":"ContainerStarted","Data":"dc64ef28d4d5791b23b5da5560b3fcff073c8568ea67a6b1effd6506b5158113"}
Jul 23 15:32:23.653074 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:32:23.653032    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-5c302e/test-model-5c302e-kserve-7d69b9749c-79gr4" event={"ID":"8bb30844-3633-4f0f-9e0f-fe5fc798dd6d","Type":"ContainerStarted","Data":"659fddbd08a45aca5325cc2f7e505775e45c0a1cd6ee360ce7d5433c13ba124c"}
Jul 23 15:32:23.670711 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:32:23.670657    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-5c302e/test-model-5c302e-kserve-7d69b9749c-79gr4" podStartSLOduration=1.670640572 podStartE2EDuration="1.670640572s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 15:32:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 15:32:23.669246699 +0000 UTC m=+2385.177120445" watchObservedRunningTime="2026-07-23 15:32:23.670640572 +0000 UTC m=+2385.178514320"
Jul 23 15:32:26.033195 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:32:26.033156    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cgwsg"]
Jul 23 15:32:32.439318 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:32:32.439277    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-5c302e/test-model-5c302e-kserve-7d69b9749c-79gr4"
Jul 23 15:32:32.439708 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:32:32.439349    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-5c302e/test-model-5c302e-kserve-7d69b9749c-79gr4"
Jul 23 15:32:32.451774 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:32:32.451745    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-5c302e/test-model-5c302e-kserve-7d69b9749c-79gr4"
Jul 23 15:32:32.713345 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:32:32.713254    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-5c302e/test-model-5c302e-kserve-7d69b9749c-79gr4"
Jul 23 15:32:40.316693 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:32:40.316580    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z4fvn_e0c66f26-06fe-4273-ba29-fa20ce89b873/ovn-acl-logging/0.log"
Jul 23 15:32:40.332682 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:32:40.324184    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z4fvn_e0c66f26-06fe-4273-ba29-fa20ce89b873/ovn-acl-logging/0.log"
Jul 23 15:32:40.926744 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:32:40.926708    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cgwsg"]
Jul 23 15:33:03.724637 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:33:03.724594    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-ec2b36/test-model-ec2b36-kserve-78f9cdb4c-7pklb"]
Jul 23 15:33:04.721773 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:33:04.721740    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-5c302e/test-model-5c302e-kserve-7d69b9749c-79gr4"]
Jul 23 15:33:07.028449 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:33:07.028408    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cgwsg"]
Jul 23 15:33:15.389824 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:33:15.389775    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ec2b36-7d9c4cfd7-7c7tm"]
Jul 23 15:33:15.390713 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:33:15.390651    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ec2b36-7d9c4cfd7-7c7tm" podUID="19775886-99a4-4d0d-887c-32098c54fb83" containerName="maas-api" containerID="cri-o://d6a2a95969bf2a3d36ee3bef6fc77c9d23b4f174fe113d7bf455cd08b3ab8f54" gracePeriod=30
Jul 23 15:33:15.632824 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:33:15.632801    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ec2b36-7d9c4cfd7-7c7tm"
Jul 23 15:33:15.729651 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:33:15.729621    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/19775886-99a4-4d0d-887c-32098c54fb83-maas-api-tls\") pod \"19775886-99a4-4d0d-887c-32098c54fb83\" (UID: \"19775886-99a4-4d0d-887c-32098c54fb83\") "
Jul 23 15:33:15.729845 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:33:15.729697    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/19775886-99a4-4d0d-887c-32098c54fb83-openshift-service-ca\") pod \"19775886-99a4-4d0d-887c-32098c54fb83\" (UID: \"19775886-99a4-4d0d-887c-32098c54fb83\") "
Jul 23 15:33:15.729845 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:33:15.729765    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g6lfv\" (UniqueName: \"kubernetes.io/projected/19775886-99a4-4d0d-887c-32098c54fb83-kube-api-access-g6lfv\") pod \"19775886-99a4-4d0d-887c-32098c54fb83\" (UID: \"19775886-99a4-4d0d-887c-32098c54fb83\") "
Jul 23 15:33:15.730081 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:33:15.730054    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19775886-99a4-4d0d-887c-32098c54fb83-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "19775886-99a4-4d0d-887c-32098c54fb83" (UID: "19775886-99a4-4d0d-887c-32098c54fb83"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 15:33:15.731851 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:33:15.731822    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19775886-99a4-4d0d-887c-32098c54fb83-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "19775886-99a4-4d0d-887c-32098c54fb83" (UID: "19775886-99a4-4d0d-887c-32098c54fb83"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 15:33:15.732060 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:33:15.732036    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19775886-99a4-4d0d-887c-32098c54fb83-kube-api-access-g6lfv" (OuterVolumeSpecName: "kube-api-access-g6lfv") pod "19775886-99a4-4d0d-887c-32098c54fb83" (UID: "19775886-99a4-4d0d-887c-32098c54fb83"). InnerVolumeSpecName "kube-api-access-g6lfv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 15:33:15.830699 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:33:15.830652    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/19775886-99a4-4d0d-887c-32098c54fb83-maas-api-tls\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Jul 23 15:33:15.830699 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:33:15.830689    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/19775886-99a4-4d0d-887c-32098c54fb83-openshift-service-ca\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Jul 23 15:33:15.830699 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:33:15.830704    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6lfv\" (UniqueName: \"kubernetes.io/projected/19775886-99a4-4d0d-887c-32098c54fb83-kube-api-access-g6lfv\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Jul 23 15:33:15.910422 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:33:15.910388    2560 generic.go:356] "Generic (PLEG): container finished" podID="19775886-99a4-4d0d-887c-32098c54fb83" containerID="d6a2a95969bf2a3d36ee3bef6fc77c9d23b4f174fe113d7bf455cd08b3ab8f54" exitCode=0
Jul 23 15:33:15.910594 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:33:15.910448    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ec2b36-7d9c4cfd7-7c7tm"
Jul 23 15:33:15.910594 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:33:15.910478    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ec2b36-7d9c4cfd7-7c7tm" event={"ID":"19775886-99a4-4d0d-887c-32098c54fb83","Type":"ContainerDied","Data":"d6a2a95969bf2a3d36ee3bef6fc77c9d23b4f174fe113d7bf455cd08b3ab8f54"}
Jul 23 15:33:15.910594 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:33:15.910569    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ec2b36-7d9c4cfd7-7c7tm" event={"ID":"19775886-99a4-4d0d-887c-32098c54fb83","Type":"ContainerDied","Data":"c4b251113ac0cd7b33e10370d4a4445f8e15b96705478265ab66c226126d20db"}
Jul 23 15:33:15.910701 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:33:15.910599    2560 scope.go:117] "RemoveContainer" containerID="d6a2a95969bf2a3d36ee3bef6fc77c9d23b4f174fe113d7bf455cd08b3ab8f54"
Jul 23 15:33:15.920490 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:33:15.920471    2560 scope.go:117] "RemoveContainer" containerID="d6a2a95969bf2a3d36ee3bef6fc77c9d23b4f174fe113d7bf455cd08b3ab8f54"
Jul 23 15:33:15.920794 ip-10-0-141-105 kubenswrapper[2560]: E0723 15:33:15.920775    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6a2a95969bf2a3d36ee3bef6fc77c9d23b4f174fe113d7bf455cd08b3ab8f54\": container with ID starting with d6a2a95969bf2a3d36ee3bef6fc77c9d23b4f174fe113d7bf455cd08b3ab8f54 not found: ID does not exist" containerID="d6a2a95969bf2a3d36ee3bef6fc77c9d23b4f174fe113d7bf455cd08b3ab8f54"
Jul 23 15:33:15.920843 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:33:15.920805    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6a2a95969bf2a3d36ee3bef6fc77c9d23b4f174fe113d7bf455cd08b3ab8f54"} err="failed to get container status \"d6a2a95969bf2a3d36ee3bef6fc77c9d23b4f174fe113d7bf455cd08b3ab8f54\": rpc error: code = NotFound desc = could not find container \"d6a2a95969bf2a3d36ee3bef6fc77c9d23b4f174fe113d7bf455cd08b3ab8f54\": container with ID starting with d6a2a95969bf2a3d36ee3bef6fc77c9d23b4f174fe113d7bf455cd08b3ab8f54 not found: ID does not exist"
Jul 23 15:33:15.932804 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:33:15.932766    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ec2b36-7d9c4cfd7-7c7tm"]
Jul 23 15:33:15.935810 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:33:15.935783    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ec2b36-7d9c4cfd7-7c7tm"]
Jul 23 15:33:17.063556 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:33:17.063520    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="19775886-99a4-4d0d-887c-32098c54fb83" path="/var/lib/kubelet/pods/19775886-99a4-4d0d-887c-32098c54fb83/volumes"
Jul 23 15:33:18.747081 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:33:18.747043    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-ec2b36/test-model-ec2b36-kserve-78f9cdb4c-7pklb" podUID="aa906dd9-9cfe-4508-9494-a07531b06d91" containerName="main" containerID="cri-o://63c1e09462e547be38e47d18cae82258d3034dc380c3eb905fb16133ef74d30c" gracePeriod=45
Jul 23 15:33:18.928486 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:33:18.928453    2560 generic.go:356] "Generic (PLEG): container finished" podID="aa906dd9-9cfe-4508-9494-a07531b06d91" containerID="63c1e09462e547be38e47d18cae82258d3034dc380c3eb905fb16133ef74d30c" exitCode=0
Jul 23 15:33:18.928660 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:33:18.928509    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ec2b36/test-model-ec2b36-kserve-78f9cdb4c-7pklb" event={"ID":"aa906dd9-9cfe-4508-9494-a07531b06d91","Type":"ContainerDied","Data":"63c1e09462e547be38e47d18cae82258d3034dc380c3eb905fb16133ef74d30c"}
Jul 23 15:33:18.997986 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:33:18.997919    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ec2b36/test-model-ec2b36-kserve-78f9cdb4c-7pklb"
Jul 23 15:33:19.162312 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:33:19.162270    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa906dd9-9cfe-4508-9494-a07531b06d91-home\") pod \"aa906dd9-9cfe-4508-9494-a07531b06d91\" (UID: \"aa906dd9-9cfe-4508-9494-a07531b06d91\") "
Jul 23 15:33:19.162312 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:33:19.162322    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cknxv\" (UniqueName: \"kubernetes.io/projected/aa906dd9-9cfe-4508-9494-a07531b06d91-kube-api-access-cknxv\") pod \"aa906dd9-9cfe-4508-9494-a07531b06d91\" (UID: \"aa906dd9-9cfe-4508-9494-a07531b06d91\") "
Jul 23 15:33:19.162561 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:33:19.162351    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa906dd9-9cfe-4508-9494-a07531b06d91-tmp-dir\") pod \"aa906dd9-9cfe-4508-9494-a07531b06d91\" (UID: \"aa906dd9-9cfe-4508-9494-a07531b06d91\") "
Jul 23 15:33:19.162561 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:33:19.162368    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa906dd9-9cfe-4508-9494-a07531b06d91-dshm\") pod \"aa906dd9-9cfe-4508-9494-a07531b06d91\" (UID: \"aa906dd9-9cfe-4508-9494-a07531b06d91\") "
Jul 23 15:33:19.162561 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:33:19.162393    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa906dd9-9cfe-4508-9494-a07531b06d91-tls-certs\") pod \"aa906dd9-9cfe-4508-9494-a07531b06d91\" (UID: \"aa906dd9-9cfe-4508-9494-a07531b06d91\") "
Jul 23 15:33:19.162561 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:33:19.162491    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa906dd9-9cfe-4508-9494-a07531b06d91-model-cache\") pod \"aa906dd9-9cfe-4508-9494-a07531b06d91\" (UID: \"aa906dd9-9cfe-4508-9494-a07531b06d91\") "
Jul 23 15:33:19.162772 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:33:19.162598    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa906dd9-9cfe-4508-9494-a07531b06d91-home" (OuterVolumeSpecName: "home") pod "aa906dd9-9cfe-4508-9494-a07531b06d91" (UID: "aa906dd9-9cfe-4508-9494-a07531b06d91"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 15:33:19.162772 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:33:19.162665    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa906dd9-9cfe-4508-9494-a07531b06d91-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "aa906dd9-9cfe-4508-9494-a07531b06d91" (UID: "aa906dd9-9cfe-4508-9494-a07531b06d91"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 15:33:19.162873 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:33:19.162838    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa906dd9-9cfe-4508-9494-a07531b06d91-model-cache" (OuterVolumeSpecName: "model-cache") pod "aa906dd9-9cfe-4508-9494-a07531b06d91" (UID: "aa906dd9-9cfe-4508-9494-a07531b06d91"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 15:33:19.162873 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:33:19.162845    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa906dd9-9cfe-4508-9494-a07531b06d91-home\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Jul 23 15:33:19.162951 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:33:19.162885    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa906dd9-9cfe-4508-9494-a07531b06d91-tmp-dir\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Jul 23 15:33:19.164811 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:33:19.164789    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa906dd9-9cfe-4508-9494-a07531b06d91-dshm" (OuterVolumeSpecName: "dshm") pod "aa906dd9-9cfe-4508-9494-a07531b06d91" (UID: "aa906dd9-9cfe-4508-9494-a07531b06d91"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 15:33:19.164936 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:33:19.164904    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa906dd9-9cfe-4508-9494-a07531b06d91-kube-api-access-cknxv" (OuterVolumeSpecName: "kube-api-access-cknxv") pod "aa906dd9-9cfe-4508-9494-a07531b06d91" (UID: "aa906dd9-9cfe-4508-9494-a07531b06d91"). InnerVolumeSpecName "kube-api-access-cknxv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 15:33:19.164936 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:33:19.164916    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa906dd9-9cfe-4508-9494-a07531b06d91-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "aa906dd9-9cfe-4508-9494-a07531b06d91" (UID: "aa906dd9-9cfe-4508-9494-a07531b06d91"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 15:33:19.263754 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:33:19.263670    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa906dd9-9cfe-4508-9494-a07531b06d91-model-cache\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Jul 23 15:33:19.263754 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:33:19.263701    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cknxv\" (UniqueName: \"kubernetes.io/projected/aa906dd9-9cfe-4508-9494-a07531b06d91-kube-api-access-cknxv\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Jul 23 15:33:19.263754 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:33:19.263714    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa906dd9-9cfe-4508-9494-a07531b06d91-dshm\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Jul 23 15:33:19.263754 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:33:19.263723    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa906dd9-9cfe-4508-9494-a07531b06d91-tls-certs\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Jul 23 15:33:19.743540 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:33:19.743488    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-5c302e/test-model-5c302e-kserve-7d69b9749c-79gr4" podUID="8bb30844-3633-4f0f-9e0f-fe5fc798dd6d" containerName="main" containerID="cri-o://659fddbd08a45aca5325cc2f7e505775e45c0a1cd6ee360ce7d5433c13ba124c" gracePeriod=45
Jul 23 15:33:19.919202 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:33:19.919176    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cgwsg"]
Jul 23 15:33:19.935726 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:33:19.935696    2560 generic.go:356] "Generic (PLEG): container finished" podID="8bb30844-3633-4f0f-9e0f-fe5fc798dd6d" containerID="659fddbd08a45aca5325cc2f7e505775e45c0a1cd6ee360ce7d5433c13ba124c" exitCode=0
Jul 23 15:33:19.935880 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:33:19.935767    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-5c302e/test-model-5c302e-kserve-7d69b9749c-79gr4" event={"ID":"8bb30844-3633-4f0f-9e0f-fe5fc798dd6d","Type":"ContainerDied","Data":"659fddbd08a45aca5325cc2f7e505775e45c0a1cd6ee360ce7d5433c13ba124c"}
Jul 23 15:33:19.937925 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:33:19.937895    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ec2b36/test-model-ec2b36-kserve-78f9cdb4c-7pklb" event={"ID":"aa906dd9-9cfe-4508-9494-a07531b06d91","Type":"ContainerDied","Data":"8ac878011f7f647ba51d2283b98dd6fefc9b20a6541c8b3136e3fbdbccddc0fb"}
Jul 23 15:33:19.938254 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:33:19.938207    2560 scope.go:117] "RemoveContainer" containerID="63c1e09462e547be38e47d18cae82258d3034dc380c3eb905fb16133ef74d30c"
Jul 23 15:33:19.938359 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:33:19.938153    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ec2b36/test-model-ec2b36-kserve-78f9cdb4c-7pklb"
Jul 23 15:33:19.962430 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:33:19.962398    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-ec2b36/test-model-ec2b36-kserve-78f9cdb4c-7pklb"]
Jul 23 15:33:19.967281 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:33:19.967252    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-ec2b36/test-model-ec2b36-kserve-78f9cdb4c-7pklb"]
Jul 23 15:33:20.007968 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:33:20.007893    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-5c302e/test-model-5c302e-kserve-7d69b9749c-79gr4"
Jul 23 15:33:20.175254 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:33:20.175204    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5m6k5\" (UniqueName: \"kubernetes.io/projected/8bb30844-3633-4f0f-9e0f-fe5fc798dd6d-kube-api-access-5m6k5\") pod \"8bb30844-3633-4f0f-9e0f-fe5fc798dd6d\" (UID: \"8bb30844-3633-4f0f-9e0f-fe5fc798dd6d\") "
Jul 23 15:33:20.175431 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:33:20.175294    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8bb30844-3633-4f0f-9e0f-fe5fc798dd6d-dshm\") pod \"8bb30844-3633-4f0f-9e0f-fe5fc798dd6d\" (UID: \"8bb30844-3633-4f0f-9e0f-fe5fc798dd6d\") "
Jul 23 15:33:20.175431 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:33:20.175344    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8bb30844-3633-4f0f-9e0f-fe5fc798dd6d-model-cache\") pod \"8bb30844-3633-4f0f-9e0f-fe5fc798dd6d\" (UID: \"8bb30844-3633-4f0f-9e0f-fe5fc798dd6d\") "
Jul 23 15:33:20.175431 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:33:20.175375    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8bb30844-3633-4f0f-9e0f-fe5fc798dd6d-tls-certs\") pod \"8bb30844-3633-4f0f-9e0f-fe5fc798dd6d\" (UID: \"8bb30844-3633-4f0f-9e0f-fe5fc798dd6d\") "
Jul 23 15:33:20.175621 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:33:20.175465    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8bb30844-3633-4f0f-9e0f-fe5fc798dd6d-home\") pod \"8bb30844-3633-4f0f-9e0f-fe5fc798dd6d\" (UID: \"8bb30844-3633-4f0f-9e0f-fe5fc798dd6d\") "
Jul 23 15:33:20.175621 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:33:20.175492    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8bb30844-3633-4f0f-9e0f-fe5fc798dd6d-tmp-dir\") pod \"8bb30844-3633-4f0f-9e0f-fe5fc798dd6d\" (UID: \"8bb30844-3633-4f0f-9e0f-fe5fc798dd6d\") "
Jul 23 15:33:20.175729 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:33:20.175636    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bb30844-3633-4f0f-9e0f-fe5fc798dd6d-model-cache" (OuterVolumeSpecName: "model-cache") pod "8bb30844-3633-4f0f-9e0f-fe5fc798dd6d" (UID: "8bb30844-3633-4f0f-9e0f-fe5fc798dd6d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 15:33:20.175784 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:33:20.175742    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8bb30844-3633-4f0f-9e0f-fe5fc798dd6d-model-cache\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Jul 23 15:33:20.175784 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:33:20.175768    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bb30844-3633-4f0f-9e0f-fe5fc798dd6d-home" (OuterVolumeSpecName: "home") pod "8bb30844-3633-4f0f-9e0f-fe5fc798dd6d" (UID: "8bb30844-3633-4f0f-9e0f-fe5fc798dd6d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 15:33:20.175901 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:33:20.175847    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bb30844-3633-4f0f-9e0f-fe5fc798dd6d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8bb30844-3633-4f0f-9e0f-fe5fc798dd6d" (UID: "8bb30844-3633-4f0f-9e0f-fe5fc798dd6d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 15:33:20.177842 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:33:20.177817    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bb30844-3633-4f0f-9e0f-fe5fc798dd6d-kube-api-access-5m6k5" (OuterVolumeSpecName: "kube-api-access-5m6k5") pod "8bb30844-3633-4f0f-9e0f-fe5fc798dd6d" (UID: "8bb30844-3633-4f0f-9e0f-fe5fc798dd6d"). InnerVolumeSpecName "kube-api-access-5m6k5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 15:33:20.178421 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:33:20.178399    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bb30844-3633-4f0f-9e0f-fe5fc798dd6d-dshm" (OuterVolumeSpecName: "dshm") pod "8bb30844-3633-4f0f-9e0f-fe5fc798dd6d" (UID: "8bb30844-3633-4f0f-9e0f-fe5fc798dd6d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 15:33:20.178421 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:33:20.178410    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bb30844-3633-4f0f-9e0f-fe5fc798dd6d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8bb30844-3633-4f0f-9e0f-fe5fc798dd6d" (UID: "8bb30844-3633-4f0f-9e0f-fe5fc798dd6d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 15:33:20.277271 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:33:20.277148    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8bb30844-3633-4f0f-9e0f-fe5fc798dd6d-home\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Jul 23 15:33:20.277271 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:33:20.277189    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8bb30844-3633-4f0f-9e0f-fe5fc798dd6d-tmp-dir\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Jul 23 15:33:20.277271 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:33:20.277207    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5m6k5\" (UniqueName: \"kubernetes.io/projected/8bb30844-3633-4f0f-9e0f-fe5fc798dd6d-kube-api-access-5m6k5\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Jul 23 15:33:20.277271 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:33:20.277219    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8bb30844-3633-4f0f-9e0f-fe5fc798dd6d-dshm\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Jul 23 15:33:20.277271 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:33:20.277248    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8bb30844-3633-4f0f-9e0f-fe5fc798dd6d-tls-certs\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Jul 23 15:33:20.944496 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:33:20.944457    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-5c302e/test-model-5c302e-kserve-7d69b9749c-79gr4" event={"ID":"8bb30844-3633-4f0f-9e0f-fe5fc798dd6d","Type":"ContainerDied","Data":"dc64ef28d4d5791b23b5da5560b3fcff073c8568ea67a6b1effd6506b5158113"}
Jul 23 15:33:20.944496 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:33:20.944468    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-5c302e/test-model-5c302e-kserve-7d69b9749c-79gr4"
Jul 23 15:33:20.944496 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:33:20.944503    2560 scope.go:117] "RemoveContainer" containerID="659fddbd08a45aca5325cc2f7e505775e45c0a1cd6ee360ce7d5433c13ba124c"
Jul 23 15:33:20.965044 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:33:20.965017    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-5c302e/test-model-5c302e-kserve-7d69b9749c-79gr4"]
Jul 23 15:33:20.968845 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:33:20.968819    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-5c302e/test-model-5c302e-kserve-7d69b9749c-79gr4"]
Jul 23 15:33:21.062546 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:33:21.062512    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8bb30844-3633-4f0f-9e0f-fe5fc798dd6d" path="/var/lib/kubelet/pods/8bb30844-3633-4f0f-9e0f-fe5fc798dd6d/volumes"
Jul 23 15:33:21.062891 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:33:21.062879    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aa906dd9-9cfe-4508-9494-a07531b06d91" path="/var/lib/kubelet/pods/aa906dd9-9cfe-4508-9494-a07531b06d91/volumes"
Jul 23 15:33:37.321485 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:33:37.321446    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5c302e-6b4ff7cf7f-x42cd"]
Jul 23 15:33:37.322252 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:33:37.321693    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5c302e-6b4ff7cf7f-x42cd" podUID="cd8266d1-d547-4b02-bc6f-4972742ca2bb" containerName="maas-api" containerID="cri-o://e3dc89cae5ecc1f1bf3fc80d6c879c5058bbac108708d1b6520595abf897fdde" gracePeriod=30
Jul 23 15:33:37.576073 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:33:37.576006    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5c302e-6b4ff7cf7f-x42cd"
Jul 23 15:33:37.633004 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:33:37.632970    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fjm8v\" (UniqueName: \"kubernetes.io/projected/cd8266d1-d547-4b02-bc6f-4972742ca2bb-kube-api-access-fjm8v\") pod \"cd8266d1-d547-4b02-bc6f-4972742ca2bb\" (UID: \"cd8266d1-d547-4b02-bc6f-4972742ca2bb\") "
Jul 23 15:33:37.633168 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:33:37.633024    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd8266d1-d547-4b02-bc6f-4972742ca2bb-openshift-service-ca\") pod \"cd8266d1-d547-4b02-bc6f-4972742ca2bb\" (UID: \"cd8266d1-d547-4b02-bc6f-4972742ca2bb\") "
Jul 23 15:33:37.633168 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:33:37.633044    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd8266d1-d547-4b02-bc6f-4972742ca2bb-maas-api-tls\") pod \"cd8266d1-d547-4b02-bc6f-4972742ca2bb\" (UID: \"cd8266d1-d547-4b02-bc6f-4972742ca2bb\") "
Jul 23 15:33:37.633440 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:33:37.633413    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd8266d1-d547-4b02-bc6f-4972742ca2bb-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "cd8266d1-d547-4b02-bc6f-4972742ca2bb" (UID: "cd8266d1-d547-4b02-bc6f-4972742ca2bb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 15:33:37.635199 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:33:37.635165    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd8266d1-d547-4b02-bc6f-4972742ca2bb-kube-api-access-fjm8v" (OuterVolumeSpecName: "kube-api-access-fjm8v") pod "cd8266d1-d547-4b02-bc6f-4972742ca2bb" (UID: "cd8266d1-d547-4b02-bc6f-4972742ca2bb"). InnerVolumeSpecName "kube-api-access-fjm8v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 15:33:37.635307 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:33:37.635245    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd8266d1-d547-4b02-bc6f-4972742ca2bb-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "cd8266d1-d547-4b02-bc6f-4972742ca2bb" (UID: "cd8266d1-d547-4b02-bc6f-4972742ca2bb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 15:33:37.734655 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:33:37.734616    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjm8v\" (UniqueName: \"kubernetes.io/projected/cd8266d1-d547-4b02-bc6f-4972742ca2bb-kube-api-access-fjm8v\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Jul 23 15:33:37.734655 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:33:37.734648    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd8266d1-d547-4b02-bc6f-4972742ca2bb-openshift-service-ca\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Jul 23 15:33:37.734655 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:33:37.734659    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd8266d1-d547-4b02-bc6f-4972742ca2bb-maas-api-tls\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Jul 23 15:33:38.025423 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:33:38.025389    2560 generic.go:356] "Generic (PLEG): container finished" podID="cd8266d1-d547-4b02-bc6f-4972742ca2bb" containerID="e3dc89cae5ecc1f1bf3fc80d6c879c5058bbac108708d1b6520595abf897fdde" exitCode=0
Jul 23 15:33:38.025594 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:33:38.025441    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5c302e-6b4ff7cf7f-x42cd"
Jul 23 15:33:38.025594 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:33:38.025451    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5c302e-6b4ff7cf7f-x42cd" event={"ID":"cd8266d1-d547-4b02-bc6f-4972742ca2bb","Type":"ContainerDied","Data":"e3dc89cae5ecc1f1bf3fc80d6c879c5058bbac108708d1b6520595abf897fdde"}
Jul 23 15:33:38.025594 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:33:38.025562    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5c302e-6b4ff7cf7f-x42cd" event={"ID":"cd8266d1-d547-4b02-bc6f-4972742ca2bb","Type":"ContainerDied","Data":"105013889ebdba6b0bc1af901cc1ebf9cf8fbac25dcd37abf0b198e801f75e9d"}
Jul 23 15:33:38.025594 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:33:38.025586    2560 scope.go:117] "RemoveContainer" containerID="e3dc89cae5ecc1f1bf3fc80d6c879c5058bbac108708d1b6520595abf897fdde"
Jul 23 15:33:38.034385 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:33:38.034368    2560 scope.go:117] "RemoveContainer" containerID="e3dc89cae5ecc1f1bf3fc80d6c879c5058bbac108708d1b6520595abf897fdde"
Jul 23 15:33:38.034638 ip-10-0-141-105 kubenswrapper[2560]: E0723 15:33:38.034620    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3dc89cae5ecc1f1bf3fc80d6c879c5058bbac108708d1b6520595abf897fdde\": container with ID starting with e3dc89cae5ecc1f1bf3fc80d6c879c5058bbac108708d1b6520595abf897fdde not found: ID does not exist" containerID="e3dc89cae5ecc1f1bf3fc80d6c879c5058bbac108708d1b6520595abf897fdde"
Jul 23 15:33:38.034678 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:33:38.034647    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3dc89cae5ecc1f1bf3fc80d6c879c5058bbac108708d1b6520595abf897fdde"} err="failed to get container status \"e3dc89cae5ecc1f1bf3fc80d6c879c5058bbac108708d1b6520595abf897fdde\": rpc error: code = NotFound desc = could not find container \"e3dc89cae5ecc1f1bf3fc80d6c879c5058bbac108708d1b6520595abf897fdde\": container with ID starting with e3dc89cae5ecc1f1bf3fc80d6c879c5058bbac108708d1b6520595abf897fdde not found: ID does not exist"
Jul 23 15:33:38.045984 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:33:38.045959    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5c302e-6b4ff7cf7f-x42cd"]
Jul 23 15:33:38.050806 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:33:38.050784    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5c302e-6b4ff7cf7f-x42cd"]
Jul 23 15:33:39.063110 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:33:39.063065    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cd8266d1-d547-4b02-bc6f-4972742ca2bb" path="/var/lib/kubelet/pods/cd8266d1-d547-4b02-bc6f-4972742ca2bb/volumes"
Jul 23 15:33:40.322518 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:33:40.322482    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cgwsg"]
Jul 23 15:33:53.040346 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:33:53.040313    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-fdfc03-55d668f8dc-v4pkt"]
Jul 23 15:33:53.040816 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:33:53.040798    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd8266d1-d547-4b02-bc6f-4972742ca2bb" containerName="maas-api"
Jul 23 15:33:53.040887 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:33:53.040820    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd8266d1-d547-4b02-bc6f-4972742ca2bb" containerName="maas-api"
Jul 23 15:33:53.040887 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:33:53.040855    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8bb30844-3633-4f0f-9e0f-fe5fc798dd6d" containerName="main"
Jul 23 15:33:53.040887 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:33:53.040865    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="8bb30844-3633-4f0f-9e0f-fe5fc798dd6d" containerName="main"
Jul 23 15:33:53.041049 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:33:53.040889    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19775886-99a4-4d0d-887c-32098c54fb83" containerName="maas-api"
Jul 23 15:33:53.041049 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:33:53.040901    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="19775886-99a4-4d0d-887c-32098c54fb83" containerName="maas-api"
Jul 23 15:33:53.041049 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:33:53.040911    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa906dd9-9cfe-4508-9494-a07531b06d91" containerName="main"
Jul 23 15:33:53.041049 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:33:53.040920    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa906dd9-9cfe-4508-9494-a07531b06d91" containerName="main"
Jul 23 15:33:53.041049 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:33:53.041003    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="19775886-99a4-4d0d-887c-32098c54fb83" containerName="maas-api"
Jul 23 15:33:53.041049 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:33:53.041016    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd8266d1-d547-4b02-bc6f-4972742ca2bb" containerName="maas-api"
Jul 23 15:33:53.041049 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:33:53.041025    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa906dd9-9cfe-4508-9494-a07531b06d91" containerName="main"
Jul 23 15:33:53.041049 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:33:53.041039    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="8bb30844-3633-4f0f-9e0f-fe5fc798dd6d" containerName="main"
Jul 23 15:33:53.043836 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:33:53.043815    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fdfc03-55d668f8dc-v4pkt"
Jul 23 15:33:53.048345 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:33:53.048317    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 23 15:33:53.049982 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:33:53.049953    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 23 15:33:53.050298 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:33:53.050278    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-rdm8l\""
Jul 23 15:33:53.050533 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:33:53.050513    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-fdfc03-serving-cert\""
Jul 23 15:33:53.063932 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:33:53.063898    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-fdfc03-55d668f8dc-v4pkt"]
Jul 23 15:33:53.170907 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:33:53.170860    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8frnj\" (UniqueName: \"kubernetes.io/projected/d502f668-a1d4-4047-bf6f-98b101565043-kube-api-access-8frnj\") pod \"maas-api-e2e-api-a-fdfc03-55d668f8dc-v4pkt\" (UID: \"d502f668-a1d4-4047-bf6f-98b101565043\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fdfc03-55d668f8dc-v4pkt"
Jul 23 15:33:53.171088 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:33:53.170941    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d502f668-a1d4-4047-bf6f-98b101565043-openshift-service-ca\") pod \"maas-api-e2e-api-a-fdfc03-55d668f8dc-v4pkt\" (UID: \"d502f668-a1d4-4047-bf6f-98b101565043\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fdfc03-55d668f8dc-v4pkt"
Jul 23 15:33:53.171088 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:33:53.170991    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d502f668-a1d4-4047-bf6f-98b101565043-maas-api-tls\") pod \"maas-api-e2e-api-a-fdfc03-55d668f8dc-v4pkt\" (UID: \"d502f668-a1d4-4047-bf6f-98b101565043\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fdfc03-55d668f8dc-v4pkt"
Jul 23 15:33:53.272415 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:33:53.272377    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8frnj\" (UniqueName: \"kubernetes.io/projected/d502f668-a1d4-4047-bf6f-98b101565043-kube-api-access-8frnj\") pod \"maas-api-e2e-api-a-fdfc03-55d668f8dc-v4pkt\" (UID: \"d502f668-a1d4-4047-bf6f-98b101565043\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fdfc03-55d668f8dc-v4pkt"
Jul 23 15:33:53.272597 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:33:53.272438    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d502f668-a1d4-4047-bf6f-98b101565043-openshift-service-ca\") pod \"maas-api-e2e-api-a-fdfc03-55d668f8dc-v4pkt\" (UID: \"d502f668-a1d4-4047-bf6f-98b101565043\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fdfc03-55d668f8dc-v4pkt"
Jul 23 15:33:53.272597 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:33:53.272484    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d502f668-a1d4-4047-bf6f-98b101565043-maas-api-tls\") pod \"maas-api-e2e-api-a-fdfc03-55d668f8dc-v4pkt\" (UID: \"d502f668-a1d4-4047-bf6f-98b101565043\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fdfc03-55d668f8dc-v4pkt"
Jul 23 15:33:53.273137 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:33:53.273115    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d502f668-a1d4-4047-bf6f-98b101565043-openshift-service-ca\") pod \"maas-api-e2e-api-a-fdfc03-55d668f8dc-v4pkt\" (UID: \"d502f668-a1d4-4047-bf6f-98b101565043\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fdfc03-55d668f8dc-v4pkt"
Jul 23 15:33:53.274995 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:33:53.274974    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d502f668-a1d4-4047-bf6f-98b101565043-maas-api-tls\") pod \"maas-api-e2e-api-a-fdfc03-55d668f8dc-v4pkt\" (UID: \"d502f668-a1d4-4047-bf6f-98b101565043\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fdfc03-55d668f8dc-v4pkt"
Jul 23 15:33:53.282008 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:33:53.281986    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8frnj\" (UniqueName: \"kubernetes.io/projected/d502f668-a1d4-4047-bf6f-98b101565043-kube-api-access-8frnj\") pod \"maas-api-e2e-api-a-fdfc03-55d668f8dc-v4pkt\" (UID: \"d502f668-a1d4-4047-bf6f-98b101565043\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fdfc03-55d668f8dc-v4pkt"
Jul 23 15:33:53.355953 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:33:53.355857    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fdfc03-55d668f8dc-v4pkt"
Jul 23 15:33:53.491902 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:33:53.491875    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-fdfc03-55d668f8dc-v4pkt"]
Jul 23 15:33:53.494502 ip-10-0-141-105 kubenswrapper[2560]: W0723 15:33:53.494473    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd502f668_a1d4_4047_bf6f_98b101565043.slice/crio-ed88a0a51ae3cab80bb1580b1bf1cd1d7d418d869e795ea2629cf2ea21bf6856 WatchSource:0}: Error finding container ed88a0a51ae3cab80bb1580b1bf1cd1d7d418d869e795ea2629cf2ea21bf6856: Status 404 returned error can't find the container with id ed88a0a51ae3cab80bb1580b1bf1cd1d7d418d869e795ea2629cf2ea21bf6856
Jul 23 15:33:54.098884 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:33:54.098848    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fdfc03-55d668f8dc-v4pkt" event={"ID":"d502f668-a1d4-4047-bf6f-98b101565043","Type":"ContainerStarted","Data":"ed88a0a51ae3cab80bb1580b1bf1cd1d7d418d869e795ea2629cf2ea21bf6856"}
Jul 23 15:33:55.104918 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:33:55.104882    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fdfc03-55d668f8dc-v4pkt" event={"ID":"d502f668-a1d4-4047-bf6f-98b101565043","Type":"ContainerStarted","Data":"000a16d56c338a17ab582f2f241d8c6baca864da2ce605daa2c24bef1dce66bc"}
Jul 23 15:33:55.105429 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:33:55.104971    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fdfc03-55d668f8dc-v4pkt"
Jul 23 15:33:55.122135 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:33:55.122080    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fdfc03-55d668f8dc-v4pkt" podStartSLOduration=1.482869012 podStartE2EDuration="2.122063473s" totalImagesPullingTime="639.194461ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 15:33:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 15:33:55.120759269 +0000 UTC m=+2476.628633018" watchObservedRunningTime="2026-07-23 15:33:55.122063473 +0000 UTC m=+2476.629937220"
Jul 23 15:33:56.024410 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:33:56.024369    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cgwsg"]
Jul 23 15:34:01.115739 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:34:01.115707    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fdfc03-55d668f8dc-v4pkt"
Jul 23 15:34:01.930478 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:34:01.930444    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8bca8c-7d699d54cf-tnz9s"]
Jul 23 15:34:01.933998 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:34:01.933974    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8bca8c-7d699d54cf-tnz9s"
Jul 23 15:34:01.937005 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:34:01.936982    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-8bca8c-serving-cert\""
Jul 23 15:34:01.943493 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:34:01.943466    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8bca8c-7d699d54cf-tnz9s"]
Jul 23 15:34:02.057199 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:34:02.057158    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66bb5708-c54e-4560-90bc-1f9ad94a4e9c-maas-api-tls\") pod \"maas-api-e2e-api-b-8bca8c-7d699d54cf-tnz9s\" (UID: \"66bb5708-c54e-4560-90bc-1f9ad94a4e9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8bca8c-7d699d54cf-tnz9s"
Jul 23 15:34:02.057199 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:34:02.057201    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66bb5708-c54e-4560-90bc-1f9ad94a4e9c-openshift-service-ca\") pod \"maas-api-e2e-api-b-8bca8c-7d699d54cf-tnz9s\" (UID: \"66bb5708-c54e-4560-90bc-1f9ad94a4e9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8bca8c-7d699d54cf-tnz9s"
Jul 23 15:34:02.057451 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:34:02.057223    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqkw6\" (UniqueName: \"kubernetes.io/projected/66bb5708-c54e-4560-90bc-1f9ad94a4e9c-kube-api-access-wqkw6\") pod \"maas-api-e2e-api-b-8bca8c-7d699d54cf-tnz9s\" (UID: \"66bb5708-c54e-4560-90bc-1f9ad94a4e9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8bca8c-7d699d54cf-tnz9s"
Jul 23 15:34:02.158584 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:34:02.158544    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66bb5708-c54e-4560-90bc-1f9ad94a4e9c-openshift-service-ca\") pod \"maas-api-e2e-api-b-8bca8c-7d699d54cf-tnz9s\" (UID: \"66bb5708-c54e-4560-90bc-1f9ad94a4e9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8bca8c-7d699d54cf-tnz9s"
Jul 23 15:34:02.158584 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:34:02.158590    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqkw6\" (UniqueName: \"kubernetes.io/projected/66bb5708-c54e-4560-90bc-1f9ad94a4e9c-kube-api-access-wqkw6\") pod \"maas-api-e2e-api-b-8bca8c-7d699d54cf-tnz9s\" (UID: \"66bb5708-c54e-4560-90bc-1f9ad94a4e9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8bca8c-7d699d54cf-tnz9s"
Jul 23 15:34:02.159109 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:34:02.158737    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66bb5708-c54e-4560-90bc-1f9ad94a4e9c-maas-api-tls\") pod \"maas-api-e2e-api-b-8bca8c-7d699d54cf-tnz9s\" (UID: \"66bb5708-c54e-4560-90bc-1f9ad94a4e9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8bca8c-7d699d54cf-tnz9s"
Jul 23 15:34:02.159510 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:34:02.159447    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66bb5708-c54e-4560-90bc-1f9ad94a4e9c-openshift-service-ca\") pod \"maas-api-e2e-api-b-8bca8c-7d699d54cf-tnz9s\" (UID: \"66bb5708-c54e-4560-90bc-1f9ad94a4e9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8bca8c-7d699d54cf-tnz9s"
Jul 23 15:34:02.161275 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:34:02.161256    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66bb5708-c54e-4560-90bc-1f9ad94a4e9c-maas-api-tls\") pod \"maas-api-e2e-api-b-8bca8c-7d699d54cf-tnz9s\" (UID: \"66bb5708-c54e-4560-90bc-1f9ad94a4e9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8bca8c-7d699d54cf-tnz9s"
Jul 23 15:34:02.169747 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:34:02.169723    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqkw6\" (UniqueName: \"kubernetes.io/projected/66bb5708-c54e-4560-90bc-1f9ad94a4e9c-kube-api-access-wqkw6\") pod \"maas-api-e2e-api-b-8bca8c-7d699d54cf-tnz9s\" (UID: \"66bb5708-c54e-4560-90bc-1f9ad94a4e9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8bca8c-7d699d54cf-tnz9s"
Jul 23 15:34:02.245723 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:34:02.245662    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8bca8c-7d699d54cf-tnz9s"
Jul 23 15:34:02.394489 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:34:02.394454    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8bca8c-7d699d54cf-tnz9s"]
Jul 23 15:34:02.398592 ip-10-0-141-105 kubenswrapper[2560]: W0723 15:34:02.398556    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66bb5708_c54e_4560_90bc_1f9ad94a4e9c.slice/crio-e5017c55e56d50b33c2f44754bd9d0c1f38e738e78f229043e3c699d9ff4cb86 WatchSource:0}: Error finding container e5017c55e56d50b33c2f44754bd9d0c1f38e738e78f229043e3c699d9ff4cb86: Status 404 returned error can't find the container with id e5017c55e56d50b33c2f44754bd9d0c1f38e738e78f229043e3c699d9ff4cb86
Jul 23 15:34:02.402317 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:34:02.402296    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 23 15:34:03.142947 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:34:03.142858    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8bca8c-7d699d54cf-tnz9s" event={"ID":"66bb5708-c54e-4560-90bc-1f9ad94a4e9c","Type":"ContainerStarted","Data":"41fe53d895345737b50463c21c1cf9e894b9e5e6604f391b3f0a9b4fc4d0bbb2"}
Jul 23 15:34:03.142947 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:34:03.142917    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8bca8c-7d699d54cf-tnz9s" event={"ID":"66bb5708-c54e-4560-90bc-1f9ad94a4e9c","Type":"ContainerStarted","Data":"e5017c55e56d50b33c2f44754bd9d0c1f38e738e78f229043e3c699d9ff4cb86"}
Jul 23 15:34:03.143141 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:34:03.142988    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8bca8c-7d699d54cf-tnz9s"
Jul 23 15:34:03.161981 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:34:03.161920    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8bca8c-7d699d54cf-tnz9s" podStartSLOduration=1.791210832 podStartE2EDuration="2.161901238s" totalImagesPullingTime="370.690406ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 15:34:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 15:34:03.16164595 +0000 UTC m=+2484.669519698" watchObservedRunningTime="2026-07-23 15:34:03.161901238 +0000 UTC m=+2484.669774987"
Jul 23 15:34:09.153242 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:34:09.153198    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8bca8c-7d699d54cf-tnz9s"
Jul 23 15:34:09.831425 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:34:09.831388    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cgwsg"]
Jul 23 15:34:30.606350 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:34:30.606267    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-fdfc03-55d668f8dc-v4pkt"]
Jul 23 15:34:30.606852 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:34:30.606820    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fdfc03-55d668f8dc-v4pkt" podUID="d502f668-a1d4-4047-bf6f-98b101565043" containerName="maas-api" containerID="cri-o://000a16d56c338a17ab582f2f241d8c6baca864da2ce605daa2c24bef1dce66bc" gracePeriod=30
Jul 23 15:34:30.852641 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:34:30.852610    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fdfc03-55d668f8dc-v4pkt"
Jul 23 15:34:30.889742 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:34:30.889667    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d502f668-a1d4-4047-bf6f-98b101565043-openshift-service-ca\") pod \"d502f668-a1d4-4047-bf6f-98b101565043\" (UID: \"d502f668-a1d4-4047-bf6f-98b101565043\") "
Jul 23 15:34:30.889890 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:34:30.889786    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d502f668-a1d4-4047-bf6f-98b101565043-maas-api-tls\") pod \"d502f668-a1d4-4047-bf6f-98b101565043\" (UID: \"d502f668-a1d4-4047-bf6f-98b101565043\") "
Jul 23 15:34:30.889890 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:34:30.889832    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8frnj\" (UniqueName: \"kubernetes.io/projected/d502f668-a1d4-4047-bf6f-98b101565043-kube-api-access-8frnj\") pod \"d502f668-a1d4-4047-bf6f-98b101565043\" (UID: \"d502f668-a1d4-4047-bf6f-98b101565043\") "
Jul 23 15:34:30.890024 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:34:30.890001    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d502f668-a1d4-4047-bf6f-98b101565043-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d502f668-a1d4-4047-bf6f-98b101565043" (UID: "d502f668-a1d4-4047-bf6f-98b101565043"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 15:34:30.892039 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:34:30.892015    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d502f668-a1d4-4047-bf6f-98b101565043-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d502f668-a1d4-4047-bf6f-98b101565043" (UID: "d502f668-a1d4-4047-bf6f-98b101565043"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 15:34:30.892153 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:34:30.892133    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d502f668-a1d4-4047-bf6f-98b101565043-kube-api-access-8frnj" (OuterVolumeSpecName: "kube-api-access-8frnj") pod "d502f668-a1d4-4047-bf6f-98b101565043" (UID: "d502f668-a1d4-4047-bf6f-98b101565043"). InnerVolumeSpecName "kube-api-access-8frnj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 15:34:30.991501 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:34:30.991461    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d502f668-a1d4-4047-bf6f-98b101565043-openshift-service-ca\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Jul 23 15:34:30.991501 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:34:30.991494    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d502f668-a1d4-4047-bf6f-98b101565043-maas-api-tls\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Jul 23 15:34:30.991501 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:34:30.991507    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8frnj\" (UniqueName: \"kubernetes.io/projected/d502f668-a1d4-4047-bf6f-98b101565043-kube-api-access-8frnj\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Jul 23 15:34:31.276603 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:34:31.276565    2560 generic.go:356] "Generic (PLEG): container finished" podID="d502f668-a1d4-4047-bf6f-98b101565043" containerID="000a16d56c338a17ab582f2f241d8c6baca864da2ce605daa2c24bef1dce66bc" exitCode=0
Jul 23 15:34:31.276804 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:34:31.276621    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fdfc03-55d668f8dc-v4pkt"
Jul 23 15:34:31.276804 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:34:31.276651    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fdfc03-55d668f8dc-v4pkt" event={"ID":"d502f668-a1d4-4047-bf6f-98b101565043","Type":"ContainerDied","Data":"000a16d56c338a17ab582f2f241d8c6baca864da2ce605daa2c24bef1dce66bc"}
Jul 23 15:34:31.276804 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:34:31.276705    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fdfc03-55d668f8dc-v4pkt" event={"ID":"d502f668-a1d4-4047-bf6f-98b101565043","Type":"ContainerDied","Data":"ed88a0a51ae3cab80bb1580b1bf1cd1d7d418d869e795ea2629cf2ea21bf6856"}
Jul 23 15:34:31.276804 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:34:31.276727    2560 scope.go:117] "RemoveContainer" containerID="000a16d56c338a17ab582f2f241d8c6baca864da2ce605daa2c24bef1dce66bc"
Jul 23 15:34:31.285209 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:34:31.285191    2560 scope.go:117] "RemoveContainer" containerID="000a16d56c338a17ab582f2f241d8c6baca864da2ce605daa2c24bef1dce66bc"
Jul 23 15:34:31.285492 ip-10-0-141-105 kubenswrapper[2560]: E0723 15:34:31.285474    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"000a16d56c338a17ab582f2f241d8c6baca864da2ce605daa2c24bef1dce66bc\": container with ID starting with 000a16d56c338a17ab582f2f241d8c6baca864da2ce605daa2c24bef1dce66bc not found: ID does not exist" containerID="000a16d56c338a17ab582f2f241d8c6baca864da2ce605daa2c24bef1dce66bc"
Jul 23 15:34:31.285560 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:34:31.285505    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"000a16d56c338a17ab582f2f241d8c6baca864da2ce605daa2c24bef1dce66bc"} err="failed to get container status \"000a16d56c338a17ab582f2f241d8c6baca864da2ce605daa2c24bef1dce66bc\": rpc error: code = NotFound desc = could not find container \"000a16d56c338a17ab582f2f241d8c6baca864da2ce605daa2c24bef1dce66bc\": container with ID starting with 000a16d56c338a17ab582f2f241d8c6baca864da2ce605daa2c24bef1dce66bc not found: ID does not exist"
Jul 23 15:34:31.294427 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:34:31.294302    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-fdfc03-55d668f8dc-v4pkt"]
Jul 23 15:34:31.295925 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:34:31.295901    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-fdfc03-55d668f8dc-v4pkt"]
Jul 23 15:34:33.062958 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:34:33.062928    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d502f668-a1d4-4047-bf6f-98b101565043" path="/var/lib/kubelet/pods/d502f668-a1d4-4047-bf6f-98b101565043/volumes"
Jul 23 15:34:37.027101 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:34:37.027065    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cgwsg"]
Jul 23 15:34:57.081878 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:34:57.081843    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8bca8c-7d699d54cf-tnz9s"]
Jul 23 15:34:57.082418 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:34:57.082085    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8bca8c-7d699d54cf-tnz9s" podUID="66bb5708-c54e-4560-90bc-1f9ad94a4e9c" containerName="maas-api" containerID="cri-o://41fe53d895345737b50463c21c1cf9e894b9e5e6604f391b3f0a9b4fc4d0bbb2" gracePeriod=30
Jul 23 15:34:57.328732 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:34:57.328709    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8bca8c-7d699d54cf-tnz9s"
Jul 23 15:34:57.399994 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:34:57.399904    2560 generic.go:356] "Generic (PLEG): container finished" podID="66bb5708-c54e-4560-90bc-1f9ad94a4e9c" containerID="41fe53d895345737b50463c21c1cf9e894b9e5e6604f391b3f0a9b4fc4d0bbb2" exitCode=0
Jul 23 15:34:57.399994 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:34:57.399944    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8bca8c-7d699d54cf-tnz9s"
Jul 23 15:34:57.400165 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:34:57.400000    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8bca8c-7d699d54cf-tnz9s" event={"ID":"66bb5708-c54e-4560-90bc-1f9ad94a4e9c","Type":"ContainerDied","Data":"41fe53d895345737b50463c21c1cf9e894b9e5e6604f391b3f0a9b4fc4d0bbb2"}
Jul 23 15:34:57.400165 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:34:57.400047    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8bca8c-7d699d54cf-tnz9s" event={"ID":"66bb5708-c54e-4560-90bc-1f9ad94a4e9c","Type":"ContainerDied","Data":"e5017c55e56d50b33c2f44754bd9d0c1f38e738e78f229043e3c699d9ff4cb86"}
Jul 23 15:34:57.400165 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:34:57.400072    2560 scope.go:117] "RemoveContainer" containerID="41fe53d895345737b50463c21c1cf9e894b9e5e6604f391b3f0a9b4fc4d0bbb2"
Jul 23 15:34:57.408571 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:34:57.408549    2560 scope.go:117] "RemoveContainer" containerID="41fe53d895345737b50463c21c1cf9e894b9e5e6604f391b3f0a9b4fc4d0bbb2"
Jul 23 15:34:57.408841 ip-10-0-141-105 kubenswrapper[2560]: E0723 15:34:57.408824    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41fe53d895345737b50463c21c1cf9e894b9e5e6604f391b3f0a9b4fc4d0bbb2\": container with ID starting with 41fe53d895345737b50463c21c1cf9e894b9e5e6604f391b3f0a9b4fc4d0bbb2 not found: ID does not exist" containerID="41fe53d895345737b50463c21c1cf9e894b9e5e6604f391b3f0a9b4fc4d0bbb2"
Jul 23 15:34:57.408919 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:34:57.408848    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41fe53d895345737b50463c21c1cf9e894b9e5e6604f391b3f0a9b4fc4d0bbb2"} err="failed to get container status \"41fe53d895345737b50463c21c1cf9e894b9e5e6604f391b3f0a9b4fc4d0bbb2\": rpc error: code = NotFound desc = could not find container \"41fe53d895345737b50463c21c1cf9e894b9e5e6604f391b3f0a9b4fc4d0bbb2\": container with ID starting with 41fe53d895345737b50463c21c1cf9e894b9e5e6604f391b3f0a9b4fc4d0bbb2 not found: ID does not exist"
Jul 23 15:34:57.437167 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:34:57.437128    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66bb5708-c54e-4560-90bc-1f9ad94a4e9c-maas-api-tls\") pod \"66bb5708-c54e-4560-90bc-1f9ad94a4e9c\" (UID: \"66bb5708-c54e-4560-90bc-1f9ad94a4e9c\") "
Jul 23 15:34:57.437167 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:34:57.437172    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wqkw6\" (UniqueName: \"kubernetes.io/projected/66bb5708-c54e-4560-90bc-1f9ad94a4e9c-kube-api-access-wqkw6\") pod \"66bb5708-c54e-4560-90bc-1f9ad94a4e9c\" (UID: \"66bb5708-c54e-4560-90bc-1f9ad94a4e9c\") "
Jul 23 15:34:57.437355 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:34:57.437219    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66bb5708-c54e-4560-90bc-1f9ad94a4e9c-openshift-service-ca\") pod \"66bb5708-c54e-4560-90bc-1f9ad94a4e9c\" (UID: \"66bb5708-c54e-4560-90bc-1f9ad94a4e9c\") "
Jul 23 15:34:57.437600 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:34:57.437573    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66bb5708-c54e-4560-90bc-1f9ad94a4e9c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "66bb5708-c54e-4560-90bc-1f9ad94a4e9c" (UID: "66bb5708-c54e-4560-90bc-1f9ad94a4e9c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 15:34:57.439397 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:34:57.439374    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66bb5708-c54e-4560-90bc-1f9ad94a4e9c-kube-api-access-wqkw6" (OuterVolumeSpecName: "kube-api-access-wqkw6") pod "66bb5708-c54e-4560-90bc-1f9ad94a4e9c" (UID: "66bb5708-c54e-4560-90bc-1f9ad94a4e9c"). InnerVolumeSpecName "kube-api-access-wqkw6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 15:34:57.439491 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:34:57.439377    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66bb5708-c54e-4560-90bc-1f9ad94a4e9c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "66bb5708-c54e-4560-90bc-1f9ad94a4e9c" (UID: "66bb5708-c54e-4560-90bc-1f9ad94a4e9c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 15:34:57.538534 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:34:57.538504    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66bb5708-c54e-4560-90bc-1f9ad94a4e9c-openshift-service-ca\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Jul 23 15:34:57.538534 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:34:57.538528    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66bb5708-c54e-4560-90bc-1f9ad94a4e9c-maas-api-tls\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Jul 23 15:34:57.538534 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:34:57.538537    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqkw6\" (UniqueName: \"kubernetes.io/projected/66bb5708-c54e-4560-90bc-1f9ad94a4e9c-kube-api-access-wqkw6\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Jul 23 15:34:57.722154 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:34:57.722108    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8bca8c-7d699d54cf-tnz9s"]
Jul 23 15:34:57.725342 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:34:57.725319    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8bca8c-7d699d54cf-tnz9s"]
Jul 23 15:34:59.061960 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:34:59.061926    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="66bb5708-c54e-4560-90bc-1f9ad94a4e9c" path="/var/lib/kubelet/pods/66bb5708-c54e-4560-90bc-1f9ad94a4e9c/volumes"
Jul 23 15:35:02.029039 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:35:02.029004    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cgwsg"]
Jul 23 15:35:12.319368 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:35:12.319334    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-d3e0e7-66b6648864-dqvjm"]
Jul 23 15:35:12.319798 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:35:12.319716    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d502f668-a1d4-4047-bf6f-98b101565043" containerName="maas-api"
Jul 23 15:35:12.319798 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:35:12.319729    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="d502f668-a1d4-4047-bf6f-98b101565043" containerName="maas-api"
Jul 23 15:35:12.319798 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:35:12.319745    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66bb5708-c54e-4560-90bc-1f9ad94a4e9c" containerName="maas-api"
Jul 23 15:35:12.319798 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:35:12.319750    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="66bb5708-c54e-4560-90bc-1f9ad94a4e9c" containerName="maas-api"
Jul 23 15:35:12.319927 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:35:12.319802    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="d502f668-a1d4-4047-bf6f-98b101565043" containerName="maas-api"
Jul 23 15:35:12.319927 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:35:12.319812    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="66bb5708-c54e-4560-90bc-1f9ad94a4e9c" containerName="maas-api"
Jul 23 15:35:12.324240 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:35:12.324206    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d3e0e7-66b6648864-dqvjm"
Jul 23 15:35:12.326848 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:35:12.326827    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-d3e0e7-serving-cert\""
Jul 23 15:35:12.327096 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:35:12.327075    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 23 15:35:12.328340 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:35:12.328323    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-rdm8l\""
Jul 23 15:35:12.328406 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:35:12.328336    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 23 15:35:12.334428 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:35:12.334405    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-d3e0e7-66b6648864-dqvjm"]
Jul 23 15:35:12.383053 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:35:12.383020    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5g497\" (UniqueName: \"kubernetes.io/projected/9289cc95-30b3-4bc8-b933-4232fc87cbf9-kube-api-access-5g497\") pod \"maas-api-e2e-shared-a-d3e0e7-66b6648864-dqvjm\" (UID: \"9289cc95-30b3-4bc8-b933-4232fc87cbf9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d3e0e7-66b6648864-dqvjm"
Jul 23 15:35:12.383261 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:35:12.383064    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9289cc95-30b3-4bc8-b933-4232fc87cbf9-maas-api-tls\") pod \"maas-api-e2e-shared-a-d3e0e7-66b6648864-dqvjm\" (UID: \"9289cc95-30b3-4bc8-b933-4232fc87cbf9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d3e0e7-66b6648864-dqvjm"
Jul 23 15:35:12.383261 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:35:12.383164    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9289cc95-30b3-4bc8-b933-4232fc87cbf9-openshift-service-ca\") pod \"maas-api-e2e-shared-a-d3e0e7-66b6648864-dqvjm\" (UID: \"9289cc95-30b3-4bc8-b933-4232fc87cbf9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d3e0e7-66b6648864-dqvjm"
Jul 23 15:35:12.483775 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:35:12.483746    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9289cc95-30b3-4bc8-b933-4232fc87cbf9-openshift-service-ca\") pod \"maas-api-e2e-shared-a-d3e0e7-66b6648864-dqvjm\" (UID: \"9289cc95-30b3-4bc8-b933-4232fc87cbf9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d3e0e7-66b6648864-dqvjm"
Jul 23 15:35:12.483933 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:35:12.483795    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5g497\" (UniqueName: \"kubernetes.io/projected/9289cc95-30b3-4bc8-b933-4232fc87cbf9-kube-api-access-5g497\") pod \"maas-api-e2e-shared-a-d3e0e7-66b6648864-dqvjm\" (UID: \"9289cc95-30b3-4bc8-b933-4232fc87cbf9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d3e0e7-66b6648864-dqvjm"
Jul 23 15:35:12.483994 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:35:12.483931    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9289cc95-30b3-4bc8-b933-4232fc87cbf9-maas-api-tls\") pod \"maas-api-e2e-shared-a-d3e0e7-66b6648864-dqvjm\" (UID: \"9289cc95-30b3-4bc8-b933-4232fc87cbf9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d3e0e7-66b6648864-dqvjm"
Jul 23 15:35:12.484074 ip-10-0-141-105 kubenswrapper[2560]: E0723 15:35:12.484057    2560 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-d3e0e7-serving-cert: secret "maas-api-e2e-shared-a-d3e0e7-serving-cert" not found
Jul 23 15:35:12.484156 ip-10-0-141-105 kubenswrapper[2560]: E0723 15:35:12.484145    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9289cc95-30b3-4bc8-b933-4232fc87cbf9-maas-api-tls podName:9289cc95-30b3-4bc8-b933-4232fc87cbf9 nodeName:}" failed. No retries permitted until 2026-07-23 15:35:12.984119639 +0000 UTC m=+2554.491993365 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9289cc95-30b3-4bc8-b933-4232fc87cbf9-maas-api-tls") pod "maas-api-e2e-shared-a-d3e0e7-66b6648864-dqvjm" (UID: "9289cc95-30b3-4bc8-b933-4232fc87cbf9") : secret "maas-api-e2e-shared-a-d3e0e7-serving-cert" not found
Jul 23 15:35:12.484473 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:35:12.484456    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9289cc95-30b3-4bc8-b933-4232fc87cbf9-openshift-service-ca\") pod \"maas-api-e2e-shared-a-d3e0e7-66b6648864-dqvjm\" (UID: \"9289cc95-30b3-4bc8-b933-4232fc87cbf9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d3e0e7-66b6648864-dqvjm"
Jul 23 15:35:12.494892 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:35:12.494865    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5g497\" (UniqueName: \"kubernetes.io/projected/9289cc95-30b3-4bc8-b933-4232fc87cbf9-kube-api-access-5g497\") pod \"maas-api-e2e-shared-a-d3e0e7-66b6648864-dqvjm\" (UID: \"9289cc95-30b3-4bc8-b933-4232fc87cbf9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d3e0e7-66b6648864-dqvjm"
Jul 23 15:35:12.988760 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:35:12.988724    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9289cc95-30b3-4bc8-b933-4232fc87cbf9-maas-api-tls\") pod \"maas-api-e2e-shared-a-d3e0e7-66b6648864-dqvjm\" (UID: \"9289cc95-30b3-4bc8-b933-4232fc87cbf9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d3e0e7-66b6648864-dqvjm"
Jul 23 15:35:12.991204 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:35:12.991184    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9289cc95-30b3-4bc8-b933-4232fc87cbf9-maas-api-tls\") pod \"maas-api-e2e-shared-a-d3e0e7-66b6648864-dqvjm\" (UID: \"9289cc95-30b3-4bc8-b933-4232fc87cbf9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d3e0e7-66b6648864-dqvjm"
Jul 23 15:35:13.235409 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:35:13.235371    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d3e0e7-66b6648864-dqvjm"
Jul 23 15:35:13.374163 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:35:13.374123    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-d3e0e7-66b6648864-dqvjm"]
Jul 23 15:35:13.377043 ip-10-0-141-105 kubenswrapper[2560]: W0723 15:35:13.377017    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9289cc95_30b3_4bc8_b933_4232fc87cbf9.slice/crio-16e959ac10affc7caabb143e8b39437838f0ccf0d81809d5388704a7625448f3 WatchSource:0}: Error finding container 16e959ac10affc7caabb143e8b39437838f0ccf0d81809d5388704a7625448f3: Status 404 returned error can't find the container with id 16e959ac10affc7caabb143e8b39437838f0ccf0d81809d5388704a7625448f3
Jul 23 15:35:13.484123 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:35:13.484079    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d3e0e7-66b6648864-dqvjm" event={"ID":"9289cc95-30b3-4bc8-b933-4232fc87cbf9","Type":"ContainerStarted","Data":"16e959ac10affc7caabb143e8b39437838f0ccf0d81809d5388704a7625448f3"}
Jul 23 15:35:14.496032 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:35:14.495994    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d3e0e7-66b6648864-dqvjm" event={"ID":"9289cc95-30b3-4bc8-b933-4232fc87cbf9","Type":"ContainerStarted","Data":"f05da802f817451f6a802c454f5699ee13099f6d61a03029aa45460a95a81149"}
Jul 23 15:35:14.496449 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:35:14.496108    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d3e0e7-66b6648864-dqvjm"
Jul 23 15:35:14.513021 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:35:14.512969    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d3e0e7-66b6648864-dqvjm" podStartSLOduration=2.10632155 podStartE2EDuration="2.512953047s" totalImagesPullingTime="406.631497ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 15:35:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 15:35:14.511730097 +0000 UTC m=+2556.019603846" watchObservedRunningTime="2026-07-23 15:35:14.512953047 +0000 UTC m=+2556.020826798"
Jul 23 15:35:17.725410 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:35:17.725374    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cgwsg"]
Jul 23 15:35:20.340595 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:35:20.340554    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-53a4f4-786d7c49bd-4wjjh"]
Jul 23 15:35:20.344132 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:35:20.344108    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-53a4f4-786d7c49bd-4wjjh"
Jul 23 15:35:20.346689 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:35:20.346664    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-53a4f4-serving-cert\""
Jul 23 15:35:20.355004 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:35:20.354977    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-53a4f4-786d7c49bd-4wjjh"]
Jul 23 15:35:20.461678 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:35:20.461643    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f084c91f-3708-4715-b7aa-b176518042bc-maas-api-tls\") pod \"maas-api-e2e-shared-b-53a4f4-786d7c49bd-4wjjh\" (UID: \"f084c91f-3708-4715-b7aa-b176518042bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-53a4f4-786d7c49bd-4wjjh"
Jul 23 15:35:20.461871 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:35:20.461685    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f084c91f-3708-4715-b7aa-b176518042bc-openshift-service-ca\") pod \"maas-api-e2e-shared-b-53a4f4-786d7c49bd-4wjjh\" (UID: \"f084c91f-3708-4715-b7aa-b176518042bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-53a4f4-786d7c49bd-4wjjh"
Jul 23 15:35:20.461871 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:35:20.461764    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tt69q\" (UniqueName: \"kubernetes.io/projected/f084c91f-3708-4715-b7aa-b176518042bc-kube-api-access-tt69q\") pod \"maas-api-e2e-shared-b-53a4f4-786d7c49bd-4wjjh\" (UID: \"f084c91f-3708-4715-b7aa-b176518042bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-53a4f4-786d7c49bd-4wjjh"
Jul 23 15:35:20.508549 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:35:20.508513    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d3e0e7-66b6648864-dqvjm"
Jul 23 15:35:20.563414 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:35:20.563365    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f084c91f-3708-4715-b7aa-b176518042bc-maas-api-tls\") pod \"maas-api-e2e-shared-b-53a4f4-786d7c49bd-4wjjh\" (UID: \"f084c91f-3708-4715-b7aa-b176518042bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-53a4f4-786d7c49bd-4wjjh"
Jul 23 15:35:20.563635 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:35:20.563432    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f084c91f-3708-4715-b7aa-b176518042bc-openshift-service-ca\") pod \"maas-api-e2e-shared-b-53a4f4-786d7c49bd-4wjjh\" (UID: \"f084c91f-3708-4715-b7aa-b176518042bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-53a4f4-786d7c49bd-4wjjh"
Jul 23 15:35:20.563635 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:35:20.563533    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tt69q\" (UniqueName: \"kubernetes.io/projected/f084c91f-3708-4715-b7aa-b176518042bc-kube-api-access-tt69q\") pod \"maas-api-e2e-shared-b-53a4f4-786d7c49bd-4wjjh\" (UID: \"f084c91f-3708-4715-b7aa-b176518042bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-53a4f4-786d7c49bd-4wjjh"
Jul 23 15:35:20.563635 ip-10-0-141-105 kubenswrapper[2560]: E0723 15:35:20.563600    2560 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-53a4f4-serving-cert: secret "maas-api-e2e-shared-b-53a4f4-serving-cert" not found
Jul 23 15:35:20.563791 ip-10-0-141-105 kubenswrapper[2560]: E0723 15:35:20.563719    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f084c91f-3708-4715-b7aa-b176518042bc-maas-api-tls podName:f084c91f-3708-4715-b7aa-b176518042bc nodeName:}" failed. No retries permitted until 2026-07-23 15:35:21.063697258 +0000 UTC m=+2562.571570984 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f084c91f-3708-4715-b7aa-b176518042bc-maas-api-tls") pod "maas-api-e2e-shared-b-53a4f4-786d7c49bd-4wjjh" (UID: "f084c91f-3708-4715-b7aa-b176518042bc") : secret "maas-api-e2e-shared-b-53a4f4-serving-cert" not found
Jul 23 15:35:20.564356 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:35:20.564330    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f084c91f-3708-4715-b7aa-b176518042bc-openshift-service-ca\") pod \"maas-api-e2e-shared-b-53a4f4-786d7c49bd-4wjjh\" (UID: \"f084c91f-3708-4715-b7aa-b176518042bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-53a4f4-786d7c49bd-4wjjh"
Jul 23 15:35:20.574789 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:35:20.574757    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tt69q\" (UniqueName: \"kubernetes.io/projected/f084c91f-3708-4715-b7aa-b176518042bc-kube-api-access-tt69q\") pod \"maas-api-e2e-shared-b-53a4f4-786d7c49bd-4wjjh\" (UID: \"f084c91f-3708-4715-b7aa-b176518042bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-53a4f4-786d7c49bd-4wjjh"
Jul 23 15:35:21.068757 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:35:21.068711    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f084c91f-3708-4715-b7aa-b176518042bc-maas-api-tls\") pod \"maas-api-e2e-shared-b-53a4f4-786d7c49bd-4wjjh\" (UID: \"f084c91f-3708-4715-b7aa-b176518042bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-53a4f4-786d7c49bd-4wjjh"
Jul 23 15:35:21.071313 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:35:21.071287    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f084c91f-3708-4715-b7aa-b176518042bc-maas-api-tls\") pod \"maas-api-e2e-shared-b-53a4f4-786d7c49bd-4wjjh\" (UID: \"f084c91f-3708-4715-b7aa-b176518042bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-53a4f4-786d7c49bd-4wjjh"
Jul 23 15:35:21.255865 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:35:21.255805    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-53a4f4-786d7c49bd-4wjjh"
Jul 23 15:35:21.398777 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:35:21.398739    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-53a4f4-786d7c49bd-4wjjh"]
Jul 23 15:35:21.401460 ip-10-0-141-105 kubenswrapper[2560]: W0723 15:35:21.401434    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf084c91f_3708_4715_b7aa_b176518042bc.slice/crio-7bb87c8c186ca2238c84332d58d574b00f1fab57bef7f6f8ebe1265a0aac22e3 WatchSource:0}: Error finding container 7bb87c8c186ca2238c84332d58d574b00f1fab57bef7f6f8ebe1265a0aac22e3: Status 404 returned error can't find the container with id 7bb87c8c186ca2238c84332d58d574b00f1fab57bef7f6f8ebe1265a0aac22e3
Jul 23 15:35:21.529821 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:35:21.529784    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-53a4f4-786d7c49bd-4wjjh" event={"ID":"f084c91f-3708-4715-b7aa-b176518042bc","Type":"ContainerStarted","Data":"7bb87c8c186ca2238c84332d58d574b00f1fab57bef7f6f8ebe1265a0aac22e3"}
Jul 23 15:35:22.535496 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:35:22.535460    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-53a4f4-786d7c49bd-4wjjh" event={"ID":"f084c91f-3708-4715-b7aa-b176518042bc","Type":"ContainerStarted","Data":"804ff05cba99377e263871b47cbd7de11e10046d1d544181417c08d14ee3a53b"}
Jul 23 15:35:22.535926 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:35:22.535600    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-53a4f4-786d7c49bd-4wjjh"
Jul 23 15:35:22.552822 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:35:22.552777    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-53a4f4-786d7c49bd-4wjjh" podStartSLOduration=1.997307336 podStartE2EDuration="2.55276308s" totalImagesPullingTime="555.455744ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 15:35:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 15:35:22.55120082 +0000 UTC m=+2564.059074569" watchObservedRunningTime="2026-07-23 15:35:22.55276308 +0000 UTC m=+2564.060636826"
Jul 23 15:35:28.546981 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:35:28.546952    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-53a4f4-786d7c49bd-4wjjh"
Jul 23 15:35:30.325914 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:35:30.325880    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cgwsg"]
Jul 23 15:35:46.624914 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:35:46.624880    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-d3e0e7/auth-test-model-d3e0e7-kserve-6776b98979-jgt2j"]
Jul 23 15:35:46.628161 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:35:46.628138    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-d3e0e7/auth-test-model-d3e0e7-kserve-6776b98979-jgt2j"
Jul 23 15:35:46.630871 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:35:46.630849    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-d3e0e7\"/\"kube-root-ca.crt\""
Jul 23 15:35:46.630987 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:35:46.630849    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-d3e0e7\"/\"openshift-service-ca.crt\""
Jul 23 15:35:46.632060 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:35:46.632037    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-d3e0e7\"/\"default-dockercfg-5h5kt\""
Jul 23 15:35:46.632173 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:35:46.632156    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-d3e0e7\"/\"auth-test-model-d3e0e7-kserve-self-signed-certs\""
Jul 23 15:35:46.637296 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:35:46.637272    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-d3e0e7/auth-test-model-d3e0e7-kserve-6776b98979-jgt2j"]
Jul 23 15:35:46.692962 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:35:46.692923    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9953708d-70b3-4283-aef9-26aba790b7d2-dshm\") pod \"auth-test-model-d3e0e7-kserve-6776b98979-jgt2j\" (UID: \"9953708d-70b3-4283-aef9-26aba790b7d2\") " pod="ai-tenant-e2e-shared-a-d3e0e7/auth-test-model-d3e0e7-kserve-6776b98979-jgt2j"
Jul 23 15:35:46.693158 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:35:46.692977    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9953708d-70b3-4283-aef9-26aba790b7d2-home\") pod \"auth-test-model-d3e0e7-kserve-6776b98979-jgt2j\" (UID: \"9953708d-70b3-4283-aef9-26aba790b7d2\") " pod="ai-tenant-e2e-shared-a-d3e0e7/auth-test-model-d3e0e7-kserve-6776b98979-jgt2j"
Jul 23 15:35:46.693158 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:35:46.693018    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9953708d-70b3-4283-aef9-26aba790b7d2-tmp-dir\") pod \"auth-test-model-d3e0e7-kserve-6776b98979-jgt2j\" (UID: \"9953708d-70b3-4283-aef9-26aba790b7d2\") " pod="ai-tenant-e2e-shared-a-d3e0e7/auth-test-model-d3e0e7-kserve-6776b98979-jgt2j"
Jul 23 15:35:46.693158 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:35:46.693037    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9953708d-70b3-4283-aef9-26aba790b7d2-model-cache\") pod \"auth-test-model-d3e0e7-kserve-6776b98979-jgt2j\" (UID: \"9953708d-70b3-4283-aef9-26aba790b7d2\") " pod="ai-tenant-e2e-shared-a-d3e0e7/auth-test-model-d3e0e7-kserve-6776b98979-jgt2j"
Jul 23 15:35:46.693158 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:35:46.693054    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9953708d-70b3-4283-aef9-26aba790b7d2-tls-certs\") pod \"auth-test-model-d3e0e7-kserve-6776b98979-jgt2j\" (UID: \"9953708d-70b3-4283-aef9-26aba790b7d2\") " pod="ai-tenant-e2e-shared-a-d3e0e7/auth-test-model-d3e0e7-kserve-6776b98979-jgt2j"
Jul 23 15:35:46.693158 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:35:46.693121    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhbmx\" (UniqueName: \"kubernetes.io/projected/9953708d-70b3-4283-aef9-26aba790b7d2-kube-api-access-mhbmx\") pod \"auth-test-model-d3e0e7-kserve-6776b98979-jgt2j\" (UID: \"9953708d-70b3-4283-aef9-26aba790b7d2\") " pod="ai-tenant-e2e-shared-a-d3e0e7/auth-test-model-d3e0e7-kserve-6776b98979-jgt2j"
Jul 23 15:35:46.793815 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:35:46.793763    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9953708d-70b3-4283-aef9-26aba790b7d2-dshm\") pod \"auth-test-model-d3e0e7-kserve-6776b98979-jgt2j\" (UID: \"9953708d-70b3-4283-aef9-26aba790b7d2\") " pod="ai-tenant-e2e-shared-a-d3e0e7/auth-test-model-d3e0e7-kserve-6776b98979-jgt2j"
Jul 23 15:35:46.793998 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:35:46.793854    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9953708d-70b3-4283-aef9-26aba790b7d2-home\") pod \"auth-test-model-d3e0e7-kserve-6776b98979-jgt2j\" (UID: \"9953708d-70b3-4283-aef9-26aba790b7d2\") " pod="ai-tenant-e2e-shared-a-d3e0e7/auth-test-model-d3e0e7-kserve-6776b98979-jgt2j"
Jul 23 15:35:46.793998 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:35:46.793913    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9953708d-70b3-4283-aef9-26aba790b7d2-tmp-dir\") pod \"auth-test-model-d3e0e7-kserve-6776b98979-jgt2j\" (UID: \"9953708d-70b3-4283-aef9-26aba790b7d2\") " pod="ai-tenant-e2e-shared-a-d3e0e7/auth-test-model-d3e0e7-kserve-6776b98979-jgt2j"
Jul 23 15:35:46.793998 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:35:46.793939    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9953708d-70b3-4283-aef9-26aba790b7d2-model-cache\") pod \"auth-test-model-d3e0e7-kserve-6776b98979-jgt2j\" (UID: \"9953708d-70b3-4283-aef9-26aba790b7d2\") " pod="ai-tenant-e2e-shared-a-d3e0e7/auth-test-model-d3e0e7-kserve-6776b98979-jgt2j"
Jul 23 15:35:46.793998 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:35:46.793965    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9953708d-70b3-4283-aef9-26aba790b7d2-tls-certs\") pod \"auth-test-model-d3e0e7-kserve-6776b98979-jgt2j\" (UID: \"9953708d-70b3-4283-aef9-26aba790b7d2\") " pod="ai-tenant-e2e-shared-a-d3e0e7/auth-test-model-d3e0e7-kserve-6776b98979-jgt2j"
Jul 23 15:35:46.793998 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:35:46.793989    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhbmx\" (UniqueName: \"kubernetes.io/projected/9953708d-70b3-4283-aef9-26aba790b7d2-kube-api-access-mhbmx\") pod \"auth-test-model-d3e0e7-kserve-6776b98979-jgt2j\" (UID: \"9953708d-70b3-4283-aef9-26aba790b7d2\") " pod="ai-tenant-e2e-shared-a-d3e0e7/auth-test-model-d3e0e7-kserve-6776b98979-jgt2j"
Jul 23 15:35:46.794293 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:35:46.794221    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9953708d-70b3-4283-aef9-26aba790b7d2-home\") pod \"auth-test-model-d3e0e7-kserve-6776b98979-jgt2j\" (UID: \"9953708d-70b3-4283-aef9-26aba790b7d2\") " pod="ai-tenant-e2e-shared-a-d3e0e7/auth-test-model-d3e0e7-kserve-6776b98979-jgt2j"
Jul 23 15:35:46.794427 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:35:46.794379    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9953708d-70b3-4283-aef9-26aba790b7d2-model-cache\") pod \"auth-test-model-d3e0e7-kserve-6776b98979-jgt2j\" (UID: \"9953708d-70b3-4283-aef9-26aba790b7d2\") " pod="ai-tenant-e2e-shared-a-d3e0e7/auth-test-model-d3e0e7-kserve-6776b98979-jgt2j"
Jul 23 15:35:46.794427 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:35:46.794402    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9953708d-70b3-4283-aef9-26aba790b7d2-tmp-dir\") pod \"auth-test-model-d3e0e7-kserve-6776b98979-jgt2j\" (UID: \"9953708d-70b3-4283-aef9-26aba790b7d2\") " pod="ai-tenant-e2e-shared-a-d3e0e7/auth-test-model-d3e0e7-kserve-6776b98979-jgt2j"
Jul 23 15:35:46.796080 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:35:46.796062    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9953708d-70b3-4283-aef9-26aba790b7d2-dshm\") pod \"auth-test-model-d3e0e7-kserve-6776b98979-jgt2j\" (UID: \"9953708d-70b3-4283-aef9-26aba790b7d2\") " pod="ai-tenant-e2e-shared-a-d3e0e7/auth-test-model-d3e0e7-kserve-6776b98979-jgt2j"
Jul 23 15:35:46.796484 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:35:46.796459    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9953708d-70b3-4283-aef9-26aba790b7d2-tls-certs\") pod \"auth-test-model-d3e0e7-kserve-6776b98979-jgt2j\" (UID: \"9953708d-70b3-4283-aef9-26aba790b7d2\") " pod="ai-tenant-e2e-shared-a-d3e0e7/auth-test-model-d3e0e7-kserve-6776b98979-jgt2j"
Jul 23 15:35:46.801878 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:35:46.801852    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhbmx\" (UniqueName: \"kubernetes.io/projected/9953708d-70b3-4283-aef9-26aba790b7d2-kube-api-access-mhbmx\") pod \"auth-test-model-d3e0e7-kserve-6776b98979-jgt2j\" (UID: \"9953708d-70b3-4283-aef9-26aba790b7d2\") " pod="ai-tenant-e2e-shared-a-d3e0e7/auth-test-model-d3e0e7-kserve-6776b98979-jgt2j"
Jul 23 15:35:46.938284 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:35:46.938153    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-d3e0e7/auth-test-model-d3e0e7-kserve-6776b98979-jgt2j"
Jul 23 15:35:47.076575 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:35:47.076549    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-d3e0e7/auth-test-model-d3e0e7-kserve-6776b98979-jgt2j"]
Jul 23 15:35:47.078913 ip-10-0-141-105 kubenswrapper[2560]: W0723 15:35:47.078885    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9953708d_70b3_4283_aef9_26aba790b7d2.slice/crio-3abeb054bb8cb954e7a7645027f70842e215cf1d278f3dee75a253b453015eb9 WatchSource:0}: Error finding container 3abeb054bb8cb954e7a7645027f70842e215cf1d278f3dee75a253b453015eb9: Status 404 returned error can't find the container with id 3abeb054bb8cb954e7a7645027f70842e215cf1d278f3dee75a253b453015eb9
Jul 23 15:35:47.659254 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:35:47.659197    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-d3e0e7/auth-test-model-d3e0e7-kserve-6776b98979-jgt2j" event={"ID":"9953708d-70b3-4283-aef9-26aba790b7d2","Type":"ContainerStarted","Data":"b0f69e8c30e394c6f40b31ee0d93261bcdcd8f90459feef8b6e961796e193850"}
Jul 23 15:35:47.659254 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:35:47.659252    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-d3e0e7/auth-test-model-d3e0e7-kserve-6776b98979-jgt2j" event={"ID":"9953708d-70b3-4283-aef9-26aba790b7d2","Type":"ContainerStarted","Data":"3abeb054bb8cb954e7a7645027f70842e215cf1d278f3dee75a253b453015eb9"}
Jul 23 15:35:47.675783 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:35:47.675733    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-d3e0e7/auth-test-model-d3e0e7-kserve-6776b98979-jgt2j" podStartSLOduration=1.675711808 podStartE2EDuration="1.675711808s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 15:35:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 15:35:47.675217586 +0000 UTC m=+2589.183091344" watchObservedRunningTime="2026-07-23 15:35:47.675711808 +0000 UTC m=+2589.183585559"
Jul 23 15:35:50.626214 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:35:50.626166    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cgwsg"]
Jul 23 15:35:56.939193 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:35:56.939155    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-d3e0e7/auth-test-model-d3e0e7-kserve-6776b98979-jgt2j"
Jul 23 15:35:56.939720 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:35:56.939246    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-d3e0e7/auth-test-model-d3e0e7-kserve-6776b98979-jgt2j"
Jul 23 15:35:56.951915 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:35:56.951883    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-d3e0e7/auth-test-model-d3e0e7-kserve-6776b98979-jgt2j"
Jul 23 15:35:57.714835 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:35:57.714807    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-d3e0e7/auth-test-model-d3e0e7-kserve-6776b98979-jgt2j"
Jul 23 15:36:10.215052 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:36:10.215017    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-53a4f4/auth-test-model-53a4f4-kserve-6bf7cb99cf-sclqn"]
Jul 23 15:36:10.219820 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:36:10.219799    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-53a4f4/auth-test-model-53a4f4-kserve-6bf7cb99cf-sclqn"
Jul 23 15:36:10.223999 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:36:10.223974    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-53a4f4\"/\"auth-test-model-53a4f4-kserve-self-signed-certs\""
Jul 23 15:36:10.224140 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:36:10.224019    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-53a4f4\"/\"kube-root-ca.crt\""
Jul 23 15:36:10.224140 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:36:10.223971    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-53a4f4\"/\"default-dockercfg-zpbqs\""
Jul 23 15:36:10.224140 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:36:10.224088    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-53a4f4\"/\"openshift-service-ca.crt\""
Jul 23 15:36:10.238745 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:36:10.238707    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-53a4f4/auth-test-model-53a4f4-kserve-6bf7cb99cf-sclqn"]
Jul 23 15:36:10.306925 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:36:10.306882    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e85b96da-ae61-4a19-b2c4-a638de169ce6-tmp-dir\") pod \"auth-test-model-53a4f4-kserve-6bf7cb99cf-sclqn\" (UID: \"e85b96da-ae61-4a19-b2c4-a638de169ce6\") " pod="ai-tenant-e2e-shared-b-53a4f4/auth-test-model-53a4f4-kserve-6bf7cb99cf-sclqn"
Jul 23 15:36:10.306925 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:36:10.306925    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nb9v\" (UniqueName: \"kubernetes.io/projected/e85b96da-ae61-4a19-b2c4-a638de169ce6-kube-api-access-2nb9v\") pod \"auth-test-model-53a4f4-kserve-6bf7cb99cf-sclqn\" (UID: \"e85b96da-ae61-4a19-b2c4-a638de169ce6\") " pod="ai-tenant-e2e-shared-b-53a4f4/auth-test-model-53a4f4-kserve-6bf7cb99cf-sclqn"
Jul 23 15:36:10.307170 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:36:10.306945    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e85b96da-ae61-4a19-b2c4-a638de169ce6-model-cache\") pod \"auth-test-model-53a4f4-kserve-6bf7cb99cf-sclqn\" (UID: \"e85b96da-ae61-4a19-b2c4-a638de169ce6\") " pod="ai-tenant-e2e-shared-b-53a4f4/auth-test-model-53a4f4-kserve-6bf7cb99cf-sclqn"
Jul 23 15:36:10.307170 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:36:10.307013    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e85b96da-ae61-4a19-b2c4-a638de169ce6-dshm\") pod \"auth-test-model-53a4f4-kserve-6bf7cb99cf-sclqn\" (UID: \"e85b96da-ae61-4a19-b2c4-a638de169ce6\") " pod="ai-tenant-e2e-shared-b-53a4f4/auth-test-model-53a4f4-kserve-6bf7cb99cf-sclqn"
Jul 23 15:36:10.307170 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:36:10.307094    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e85b96da-ae61-4a19-b2c4-a638de169ce6-tls-certs\") pod \"auth-test-model-53a4f4-kserve-6bf7cb99cf-sclqn\" (UID: \"e85b96da-ae61-4a19-b2c4-a638de169ce6\") " pod="ai-tenant-e2e-shared-b-53a4f4/auth-test-model-53a4f4-kserve-6bf7cb99cf-sclqn"
Jul 23 15:36:10.307170 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:36:10.307120    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e85b96da-ae61-4a19-b2c4-a638de169ce6-home\") pod \"auth-test-model-53a4f4-kserve-6bf7cb99cf-sclqn\" (UID: \"e85b96da-ae61-4a19-b2c4-a638de169ce6\") " pod="ai-tenant-e2e-shared-b-53a4f4/auth-test-model-53a4f4-kserve-6bf7cb99cf-sclqn"
Jul 23 15:36:10.407890 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:36:10.407845    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e85b96da-ae61-4a19-b2c4-a638de169ce6-tmp-dir\") pod \"auth-test-model-53a4f4-kserve-6bf7cb99cf-sclqn\" (UID: \"e85b96da-ae61-4a19-b2c4-a638de169ce6\") " pod="ai-tenant-e2e-shared-b-53a4f4/auth-test-model-53a4f4-kserve-6bf7cb99cf-sclqn"
Jul 23 15:36:10.407890 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:36:10.407893    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nb9v\" (UniqueName: \"kubernetes.io/projected/e85b96da-ae61-4a19-b2c4-a638de169ce6-kube-api-access-2nb9v\") pod \"auth-test-model-53a4f4-kserve-6bf7cb99cf-sclqn\" (UID: \"e85b96da-ae61-4a19-b2c4-a638de169ce6\") " pod="ai-tenant-e2e-shared-b-53a4f4/auth-test-model-53a4f4-kserve-6bf7cb99cf-sclqn"
Jul 23 15:36:10.408155 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:36:10.408013    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e85b96da-ae61-4a19-b2c4-a638de169ce6-model-cache\") pod \"auth-test-model-53a4f4-kserve-6bf7cb99cf-sclqn\" (UID: \"e85b96da-ae61-4a19-b2c4-a638de169ce6\") " pod="ai-tenant-e2e-shared-b-53a4f4/auth-test-model-53a4f4-kserve-6bf7cb99cf-sclqn"
Jul 23 15:36:10.408155 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:36:10.408050    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e85b96da-ae61-4a19-b2c4-a638de169ce6-dshm\") pod \"auth-test-model-53a4f4-kserve-6bf7cb99cf-sclqn\" (UID: \"e85b96da-ae61-4a19-b2c4-a638de169ce6\") " pod="ai-tenant-e2e-shared-b-53a4f4/auth-test-model-53a4f4-kserve-6bf7cb99cf-sclqn"
Jul 23 15:36:10.408155 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:36:10.408098    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e85b96da-ae61-4a19-b2c4-a638de169ce6-tls-certs\") pod \"auth-test-model-53a4f4-kserve-6bf7cb99cf-sclqn\" (UID: \"e85b96da-ae61-4a19-b2c4-a638de169ce6\") " pod="ai-tenant-e2e-shared-b-53a4f4/auth-test-model-53a4f4-kserve-6bf7cb99cf-sclqn"
Jul 23 15:36:10.408155 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:36:10.408123    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e85b96da-ae61-4a19-b2c4-a638de169ce6-home\") pod \"auth-test-model-53a4f4-kserve-6bf7cb99cf-sclqn\" (UID: \"e85b96da-ae61-4a19-b2c4-a638de169ce6\") " pod="ai-tenant-e2e-shared-b-53a4f4/auth-test-model-53a4f4-kserve-6bf7cb99cf-sclqn"
Jul 23 15:36:10.408390 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:36:10.408338    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e85b96da-ae61-4a19-b2c4-a638de169ce6-tmp-dir\") pod \"auth-test-model-53a4f4-kserve-6bf7cb99cf-sclqn\" (UID: \"e85b96da-ae61-4a19-b2c4-a638de169ce6\") " pod="ai-tenant-e2e-shared-b-53a4f4/auth-test-model-53a4f4-kserve-6bf7cb99cf-sclqn"
Jul 23 15:36:10.408390 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:36:10.408381    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e85b96da-ae61-4a19-b2c4-a638de169ce6-model-cache\") pod \"auth-test-model-53a4f4-kserve-6bf7cb99cf-sclqn\" (UID: \"e85b96da-ae61-4a19-b2c4-a638de169ce6\") " pod="ai-tenant-e2e-shared-b-53a4f4/auth-test-model-53a4f4-kserve-6bf7cb99cf-sclqn"
Jul 23 15:36:10.408587 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:36:10.408563    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e85b96da-ae61-4a19-b2c4-a638de169ce6-home\") pod \"auth-test-model-53a4f4-kserve-6bf7cb99cf-sclqn\" (UID: \"e85b96da-ae61-4a19-b2c4-a638de169ce6\") " pod="ai-tenant-e2e-shared-b-53a4f4/auth-test-model-53a4f4-kserve-6bf7cb99cf-sclqn"
Jul 23 15:36:10.410608 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:36:10.410579    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e85b96da-ae61-4a19-b2c4-a638de169ce6-dshm\") pod \"auth-test-model-53a4f4-kserve-6bf7cb99cf-sclqn\" (UID: \"e85b96da-ae61-4a19-b2c4-a638de169ce6\") " pod="ai-tenant-e2e-shared-b-53a4f4/auth-test-model-53a4f4-kserve-6bf7cb99cf-sclqn"
Jul 23 15:36:10.410807 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:36:10.410791    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e85b96da-ae61-4a19-b2c4-a638de169ce6-tls-certs\") pod \"auth-test-model-53a4f4-kserve-6bf7cb99cf-sclqn\" (UID: \"e85b96da-ae61-4a19-b2c4-a638de169ce6\") " pod="ai-tenant-e2e-shared-b-53a4f4/auth-test-model-53a4f4-kserve-6bf7cb99cf-sclqn"
Jul 23 15:36:10.416959 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:36:10.416926    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nb9v\" (UniqueName: \"kubernetes.io/projected/e85b96da-ae61-4a19-b2c4-a638de169ce6-kube-api-access-2nb9v\") pod \"auth-test-model-53a4f4-kserve-6bf7cb99cf-sclqn\" (UID: \"e85b96da-ae61-4a19-b2c4-a638de169ce6\") " pod="ai-tenant-e2e-shared-b-53a4f4/auth-test-model-53a4f4-kserve-6bf7cb99cf-sclqn"
Jul 23 15:36:10.529549 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:36:10.529445    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-53a4f4/auth-test-model-53a4f4-kserve-6bf7cb99cf-sclqn"
Jul 23 15:36:10.673505 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:36:10.673480    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-53a4f4/auth-test-model-53a4f4-kserve-6bf7cb99cf-sclqn"]
Jul 23 15:36:10.676534 ip-10-0-141-105 kubenswrapper[2560]: W0723 15:36:10.676503    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode85b96da_ae61_4a19_b2c4_a638de169ce6.slice/crio-674da9593514018705b1c29602335e5c92959220c0a9ac4d9ee7d0f8d98c7be6 WatchSource:0}: Error finding container 674da9593514018705b1c29602335e5c92959220c0a9ac4d9ee7d0f8d98c7be6: Status 404 returned error can't find the container with id 674da9593514018705b1c29602335e5c92959220c0a9ac4d9ee7d0f8d98c7be6
Jul 23 15:36:10.766204 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:36:10.766155    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-53a4f4/auth-test-model-53a4f4-kserve-6bf7cb99cf-sclqn" event={"ID":"e85b96da-ae61-4a19-b2c4-a638de169ce6","Type":"ContainerStarted","Data":"a31d202e8a62432a66d1d558b6613692c1f827b6ffd32590b13499c841e90261"}
Jul 23 15:36:10.766389 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:36:10.766213    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-53a4f4/auth-test-model-53a4f4-kserve-6bf7cb99cf-sclqn" event={"ID":"e85b96da-ae61-4a19-b2c4-a638de169ce6","Type":"ContainerStarted","Data":"674da9593514018705b1c29602335e5c92959220c0a9ac4d9ee7d0f8d98c7be6"}
Jul 23 15:36:10.782761 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:36:10.782632    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-53a4f4/auth-test-model-53a4f4-kserve-6bf7cb99cf-sclqn" podStartSLOduration=0.782615758 podStartE2EDuration="782.615758ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 15:36:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 15:36:10.781803024 +0000 UTC m=+2612.289676772" watchObservedRunningTime="2026-07-23 15:36:10.782615758 +0000 UTC m=+2612.290489506"
Jul 23 15:36:13.820093 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:36:13.820052    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cgwsg"]
Jul 23 15:36:20.529893 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:36:20.529852    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-53a4f4/auth-test-model-53a4f4-kserve-6bf7cb99cf-sclqn"
Jul 23 15:36:20.530457 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:36:20.529928    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-53a4f4/auth-test-model-53a4f4-kserve-6bf7cb99cf-sclqn"
Jul 23 15:36:20.542502 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:36:20.542470    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-53a4f4/auth-test-model-53a4f4-kserve-6bf7cb99cf-sclqn"
Jul 23 15:36:20.825402 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:36:20.825318    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-53a4f4/auth-test-model-53a4f4-kserve-6bf7cb99cf-sclqn"
Jul 23 15:36:27.621480 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:36:27.621440    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cgwsg"]
Jul 23 15:36:47.428008 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:36:47.427966    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cgwsg"]
Jul 23 15:37:06.023062 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:37:06.023026    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cgwsg"]
Jul 23 15:37:20.127741 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:37:20.127697    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cgwsg"]
Jul 23 15:37:37.921825 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:37:37.921782    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cgwsg"]
Jul 23 15:37:40.346393 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:37:40.346288    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z4fvn_e0c66f26-06fe-4273-ba29-fa20ce89b873/ovn-acl-logging/0.log"
Jul 23 15:37:40.355649 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:37:40.355629    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z4fvn_e0c66f26-06fe-4273-ba29-fa20ce89b873/ovn-acl-logging/0.log"
Jul 23 15:37:57.437257 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:37:57.437206    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cgwsg"]
Jul 23 15:38:13.428192 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:38:13.428154    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-d3e0e7/auth-test-model-d3e0e7-kserve-6776b98979-jgt2j"]
Jul 23 15:38:14.471538 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:38:14.471494    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-53a4f4/auth-test-model-53a4f4-kserve-6bf7cb99cf-sclqn"]
Jul 23 15:38:16.321583 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:38:16.321545    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cgwsg"]
Jul 23 15:38:28.449309 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:38:28.449263    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-d3e0e7/auth-test-model-d3e0e7-kserve-6776b98979-jgt2j" podUID="9953708d-70b3-4283-aef9-26aba790b7d2" containerName="main" containerID="cri-o://b0f69e8c30e394c6f40b31ee0d93261bcdcd8f90459feef8b6e961796e193850" gracePeriod=45
Jul 23 15:38:28.703701 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:38:28.703633    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-d3e0e7/auth-test-model-d3e0e7-kserve-6776b98979-jgt2j"
Jul 23 15:38:28.788333 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:38:28.788292    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9953708d-70b3-4283-aef9-26aba790b7d2-model-cache\") pod \"9953708d-70b3-4283-aef9-26aba790b7d2\" (UID: \"9953708d-70b3-4283-aef9-26aba790b7d2\") "
Jul 23 15:38:28.788333 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:38:28.788342    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9953708d-70b3-4283-aef9-26aba790b7d2-tls-certs\") pod \"9953708d-70b3-4283-aef9-26aba790b7d2\" (UID: \"9953708d-70b3-4283-aef9-26aba790b7d2\") "
Jul 23 15:38:28.788599 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:38:28.788364    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mhbmx\" (UniqueName: \"kubernetes.io/projected/9953708d-70b3-4283-aef9-26aba790b7d2-kube-api-access-mhbmx\") pod \"9953708d-70b3-4283-aef9-26aba790b7d2\" (UID: \"9953708d-70b3-4283-aef9-26aba790b7d2\") "
Jul 23 15:38:28.788599 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:38:28.788429    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9953708d-70b3-4283-aef9-26aba790b7d2-dshm\") pod \"9953708d-70b3-4283-aef9-26aba790b7d2\" (UID: \"9953708d-70b3-4283-aef9-26aba790b7d2\") "
Jul 23 15:38:28.788599 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:38:28.788463    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9953708d-70b3-4283-aef9-26aba790b7d2-home\") pod \"9953708d-70b3-4283-aef9-26aba790b7d2\" (UID: \"9953708d-70b3-4283-aef9-26aba790b7d2\") "
Jul 23 15:38:28.788599 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:38:28.788516    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9953708d-70b3-4283-aef9-26aba790b7d2-tmp-dir\") pod \"9953708d-70b3-4283-aef9-26aba790b7d2\" (UID: \"9953708d-70b3-4283-aef9-26aba790b7d2\") "
Jul 23 15:38:28.788815 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:38:28.788621    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9953708d-70b3-4283-aef9-26aba790b7d2-model-cache" (OuterVolumeSpecName: "model-cache") pod "9953708d-70b3-4283-aef9-26aba790b7d2" (UID: "9953708d-70b3-4283-aef9-26aba790b7d2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 15:38:28.788815 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:38:28.788735    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9953708d-70b3-4283-aef9-26aba790b7d2-home" (OuterVolumeSpecName: "home") pod "9953708d-70b3-4283-aef9-26aba790b7d2" (UID: "9953708d-70b3-4283-aef9-26aba790b7d2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 15:38:28.788815 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:38:28.788796    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9953708d-70b3-4283-aef9-26aba790b7d2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9953708d-70b3-4283-aef9-26aba790b7d2" (UID: "9953708d-70b3-4283-aef9-26aba790b7d2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 15:38:28.788926 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:38:28.788856    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9953708d-70b3-4283-aef9-26aba790b7d2-home\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Jul 23 15:38:28.788926 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:38:28.788879    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9953708d-70b3-4283-aef9-26aba790b7d2-model-cache\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Jul 23 15:38:28.790793 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:38:28.790768    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9953708d-70b3-4283-aef9-26aba790b7d2-dshm" (OuterVolumeSpecName: "dshm") pod "9953708d-70b3-4283-aef9-26aba790b7d2" (UID: "9953708d-70b3-4283-aef9-26aba790b7d2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 15:38:28.790793 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:38:28.790785    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9953708d-70b3-4283-aef9-26aba790b7d2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9953708d-70b3-4283-aef9-26aba790b7d2" (UID: "9953708d-70b3-4283-aef9-26aba790b7d2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 15:38:28.790961 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:38:28.790768    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9953708d-70b3-4283-aef9-26aba790b7d2-kube-api-access-mhbmx" (OuterVolumeSpecName: "kube-api-access-mhbmx") pod "9953708d-70b3-4283-aef9-26aba790b7d2" (UID: "9953708d-70b3-4283-aef9-26aba790b7d2"). InnerVolumeSpecName "kube-api-access-mhbmx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 15:38:28.889964 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:38:28.889925    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9953708d-70b3-4283-aef9-26aba790b7d2-tmp-dir\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Jul 23 15:38:28.889964 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:38:28.889957    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9953708d-70b3-4283-aef9-26aba790b7d2-tls-certs\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Jul 23 15:38:28.889964 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:38:28.889969    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhbmx\" (UniqueName: \"kubernetes.io/projected/9953708d-70b3-4283-aef9-26aba790b7d2-kube-api-access-mhbmx\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Jul 23 15:38:28.890215 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:38:28.889979    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9953708d-70b3-4283-aef9-26aba790b7d2-dshm\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Jul 23 15:38:29.426067 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:38:29.425963    2560 generic.go:356] "Generic (PLEG): container finished" podID="9953708d-70b3-4283-aef9-26aba790b7d2" containerID="b0f69e8c30e394c6f40b31ee0d93261bcdcd8f90459feef8b6e961796e193850" exitCode=0
Jul 23 15:38:29.426067 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:38:29.426030    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-d3e0e7/auth-test-model-d3e0e7-kserve-6776b98979-jgt2j"
Jul 23 15:38:29.426067 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:38:29.426052    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-d3e0e7/auth-test-model-d3e0e7-kserve-6776b98979-jgt2j" event={"ID":"9953708d-70b3-4283-aef9-26aba790b7d2","Type":"ContainerDied","Data":"b0f69e8c30e394c6f40b31ee0d93261bcdcd8f90459feef8b6e961796e193850"}
Jul 23 15:38:29.426323 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:38:29.426102    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-d3e0e7/auth-test-model-d3e0e7-kserve-6776b98979-jgt2j" event={"ID":"9953708d-70b3-4283-aef9-26aba790b7d2","Type":"ContainerDied","Data":"3abeb054bb8cb954e7a7645027f70842e215cf1d278f3dee75a253b453015eb9"}
Jul 23 15:38:29.426323 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:38:29.426124    2560 scope.go:117] "RemoveContainer" containerID="b0f69e8c30e394c6f40b31ee0d93261bcdcd8f90459feef8b6e961796e193850"
Jul 23 15:38:29.435351 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:38:29.435323    2560 scope.go:117] "RemoveContainer" containerID="b0f69e8c30e394c6f40b31ee0d93261bcdcd8f90459feef8b6e961796e193850"
Jul 23 15:38:29.435680 ip-10-0-141-105 kubenswrapper[2560]: E0723 15:38:29.435657    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0f69e8c30e394c6f40b31ee0d93261bcdcd8f90459feef8b6e961796e193850\": container with ID starting with b0f69e8c30e394c6f40b31ee0d93261bcdcd8f90459feef8b6e961796e193850 not found: ID does not exist" containerID="b0f69e8c30e394c6f40b31ee0d93261bcdcd8f90459feef8b6e961796e193850"
Jul 23 15:38:29.435831 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:38:29.435688    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0f69e8c30e394c6f40b31ee0d93261bcdcd8f90459feef8b6e961796e193850"} err="failed to get container status \"b0f69e8c30e394c6f40b31ee0d93261bcdcd8f90459feef8b6e961796e193850\": rpc error: code = NotFound desc = could not find container \"b0f69e8c30e394c6f40b31ee0d93261bcdcd8f90459feef8b6e961796e193850\": container with ID starting with b0f69e8c30e394c6f40b31ee0d93261bcdcd8f90459feef8b6e961796e193850 not found: ID does not exist"
Jul 23 15:38:29.443552 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:38:29.443525    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-d3e0e7/auth-test-model-d3e0e7-kserve-6776b98979-jgt2j"]
Jul 23 15:38:29.447040 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:38:29.447015    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-d3e0e7/auth-test-model-d3e0e7-kserve-6776b98979-jgt2j"]
Jul 23 15:38:29.494291 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:38:29.494222    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-53a4f4/auth-test-model-53a4f4-kserve-6bf7cb99cf-sclqn" podUID="e85b96da-ae61-4a19-b2c4-a638de169ce6" containerName="main" containerID="cri-o://a31d202e8a62432a66d1d558b6613692c1f827b6ffd32590b13499c841e90261" gracePeriod=45
Jul 23 15:38:29.752224 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:38:29.752196    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-53a4f4/auth-test-model-53a4f4-kserve-6bf7cb99cf-sclqn"
Jul 23 15:38:29.897414 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:38:29.897383    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e85b96da-ae61-4a19-b2c4-a638de169ce6-tmp-dir\") pod \"e85b96da-ae61-4a19-b2c4-a638de169ce6\" (UID: \"e85b96da-ae61-4a19-b2c4-a638de169ce6\") "
Jul 23 15:38:29.897414 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:38:29.897432    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e85b96da-ae61-4a19-b2c4-a638de169ce6-tls-certs\") pod \"e85b96da-ae61-4a19-b2c4-a638de169ce6\" (UID: \"e85b96da-ae61-4a19-b2c4-a638de169ce6\") "
Jul 23 15:38:29.897685 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:38:29.897453    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e85b96da-ae61-4a19-b2c4-a638de169ce6-dshm\") pod \"e85b96da-ae61-4a19-b2c4-a638de169ce6\" (UID: \"e85b96da-ae61-4a19-b2c4-a638de169ce6\") "
Jul 23 15:38:29.897685 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:38:29.897479    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e85b96da-ae61-4a19-b2c4-a638de169ce6-model-cache\") pod \"e85b96da-ae61-4a19-b2c4-a638de169ce6\" (UID: \"e85b96da-ae61-4a19-b2c4-a638de169ce6\") "
Jul 23 15:38:29.897685 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:38:29.897496    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e85b96da-ae61-4a19-b2c4-a638de169ce6-home\") pod \"e85b96da-ae61-4a19-b2c4-a638de169ce6\" (UID: \"e85b96da-ae61-4a19-b2c4-a638de169ce6\") "
Jul 23 15:38:29.897685 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:38:29.897556    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2nb9v\" (UniqueName: \"kubernetes.io/projected/e85b96da-ae61-4a19-b2c4-a638de169ce6-kube-api-access-2nb9v\") pod \"e85b96da-ae61-4a19-b2c4-a638de169ce6\" (UID: \"e85b96da-ae61-4a19-b2c4-a638de169ce6\") "
Jul 23 15:38:29.897892 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:38:29.897759    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e85b96da-ae61-4a19-b2c4-a638de169ce6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e85b96da-ae61-4a19-b2c4-a638de169ce6" (UID: "e85b96da-ae61-4a19-b2c4-a638de169ce6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 15:38:29.897892 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:38:29.897783    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e85b96da-ae61-4a19-b2c4-a638de169ce6-model-cache" (OuterVolumeSpecName: "model-cache") pod "e85b96da-ae61-4a19-b2c4-a638de169ce6" (UID: "e85b96da-ae61-4a19-b2c4-a638de169ce6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 15:38:29.897892 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:38:29.897795    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e85b96da-ae61-4a19-b2c4-a638de169ce6-home" (OuterVolumeSpecName: "home") pod "e85b96da-ae61-4a19-b2c4-a638de169ce6" (UID: "e85b96da-ae61-4a19-b2c4-a638de169ce6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 15:38:29.899992 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:38:29.899962    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e85b96da-ae61-4a19-b2c4-a638de169ce6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e85b96da-ae61-4a19-b2c4-a638de169ce6" (UID: "e85b96da-ae61-4a19-b2c4-a638de169ce6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 15:38:29.900092 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:38:29.899985    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e85b96da-ae61-4a19-b2c4-a638de169ce6-kube-api-access-2nb9v" (OuterVolumeSpecName: "kube-api-access-2nb9v") pod "e85b96da-ae61-4a19-b2c4-a638de169ce6" (UID: "e85b96da-ae61-4a19-b2c4-a638de169ce6"). InnerVolumeSpecName "kube-api-access-2nb9v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 15:38:29.900092 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:38:29.900046    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e85b96da-ae61-4a19-b2c4-a638de169ce6-dshm" (OuterVolumeSpecName: "dshm") pod "e85b96da-ae61-4a19-b2c4-a638de169ce6" (UID: "e85b96da-ae61-4a19-b2c4-a638de169ce6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 15:38:29.998896 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:38:29.998790    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e85b96da-ae61-4a19-b2c4-a638de169ce6-model-cache\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Jul 23 15:38:29.998896 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:38:29.998828    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e85b96da-ae61-4a19-b2c4-a638de169ce6-home\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Jul 23 15:38:29.998896 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:38:29.998841    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2nb9v\" (UniqueName: \"kubernetes.io/projected/e85b96da-ae61-4a19-b2c4-a638de169ce6-kube-api-access-2nb9v\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Jul 23 15:38:29.998896 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:38:29.998882    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e85b96da-ae61-4a19-b2c4-a638de169ce6-tmp-dir\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Jul 23 15:38:29.998896 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:38:29.998899    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e85b96da-ae61-4a19-b2c4-a638de169ce6-tls-certs\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Jul 23 15:38:29.999178 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:38:29.998911    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e85b96da-ae61-4a19-b2c4-a638de169ce6-dshm\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Jul 23 15:38:30.433048 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:38:30.432946    2560 generic.go:356] "Generic (PLEG): container finished" podID="e85b96da-ae61-4a19-b2c4-a638de169ce6" containerID="a31d202e8a62432a66d1d558b6613692c1f827b6ffd32590b13499c841e90261" exitCode=0
Jul 23 15:38:30.433048 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:38:30.432986    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-53a4f4/auth-test-model-53a4f4-kserve-6bf7cb99cf-sclqn"
Jul 23 15:38:30.433048 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:38:30.432991    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-53a4f4/auth-test-model-53a4f4-kserve-6bf7cb99cf-sclqn" event={"ID":"e85b96da-ae61-4a19-b2c4-a638de169ce6","Type":"ContainerDied","Data":"a31d202e8a62432a66d1d558b6613692c1f827b6ffd32590b13499c841e90261"}
Jul 23 15:38:30.433048 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:38:30.433030    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-53a4f4/auth-test-model-53a4f4-kserve-6bf7cb99cf-sclqn" event={"ID":"e85b96da-ae61-4a19-b2c4-a638de169ce6","Type":"ContainerDied","Data":"674da9593514018705b1c29602335e5c92959220c0a9ac4d9ee7d0f8d98c7be6"}
Jul 23 15:38:30.433048 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:38:30.433052    2560 scope.go:117] "RemoveContainer" containerID="a31d202e8a62432a66d1d558b6613692c1f827b6ffd32590b13499c841e90261"
Jul 23 15:38:30.442427 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:38:30.442408    2560 scope.go:117] "RemoveContainer" containerID="a31d202e8a62432a66d1d558b6613692c1f827b6ffd32590b13499c841e90261"
Jul 23 15:38:30.442724 ip-10-0-141-105 kubenswrapper[2560]: E0723 15:38:30.442705    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a31d202e8a62432a66d1d558b6613692c1f827b6ffd32590b13499c841e90261\": container with ID starting with a31d202e8a62432a66d1d558b6613692c1f827b6ffd32590b13499c841e90261 not found: ID does not exist" containerID="a31d202e8a62432a66d1d558b6613692c1f827b6ffd32590b13499c841e90261"
Jul 23 15:38:30.442777 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:38:30.442738    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a31d202e8a62432a66d1d558b6613692c1f827b6ffd32590b13499c841e90261"} err="failed to get container status \"a31d202e8a62432a66d1d558b6613692c1f827b6ffd32590b13499c841e90261\": rpc error: code = NotFound desc = could not find container \"a31d202e8a62432a66d1d558b6613692c1f827b6ffd32590b13499c841e90261\": container with ID starting with a31d202e8a62432a66d1d558b6613692c1f827b6ffd32590b13499c841e90261 not found: ID does not exist"
Jul 23 15:38:30.453639 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:38:30.453600    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-53a4f4/auth-test-model-53a4f4-kserve-6bf7cb99cf-sclqn"]
Jul 23 15:38:30.457169 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:38:30.457139    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-53a4f4/auth-test-model-53a4f4-kserve-6bf7cb99cf-sclqn"]
Jul 23 15:38:31.062538 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:38:31.062507    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9953708d-70b3-4283-aef9-26aba790b7d2" path="/var/lib/kubelet/pods/9953708d-70b3-4283-aef9-26aba790b7d2/volumes"
Jul 23 15:38:31.062925 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:38:31.062867    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e85b96da-ae61-4a19-b2c4-a638de169ce6" path="/var/lib/kubelet/pods/e85b96da-ae61-4a19-b2c4-a638de169ce6/volumes"
Jul 23 15:38:31.319577 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:38:31.319472    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cgwsg"]
Jul 23 15:38:42.931282 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:38:42.931247    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-53a4f4/sub-test-model-53a4f4-kserve-57dd967d5-dxbn9"]
Jul 23 15:38:42.931653 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:38:42.931611    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9953708d-70b3-4283-aef9-26aba790b7d2" containerName="main"
Jul 23 15:38:42.931653 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:38:42.931621    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="9953708d-70b3-4283-aef9-26aba790b7d2" containerName="main"
Jul 23 15:38:42.931653 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:38:42.931645    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e85b96da-ae61-4a19-b2c4-a638de169ce6" containerName="main"
Jul 23 15:38:42.931653 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:38:42.931651    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="e85b96da-ae61-4a19-b2c4-a638de169ce6" containerName="main"
Jul 23 15:38:42.931785 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:38:42.931707    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="9953708d-70b3-4283-aef9-26aba790b7d2" containerName="main"
Jul 23 15:38:42.931785 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:38:42.931717    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="e85b96da-ae61-4a19-b2c4-a638de169ce6" containerName="main"
Jul 23 15:38:42.936314 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:38:42.936295    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-53a4f4/sub-test-model-53a4f4-kserve-57dd967d5-dxbn9"
Jul 23 15:38:42.939185 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:38:42.939157    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-53a4f4\"/\"openshift-service-ca.crt\""
Jul 23 15:38:42.940458 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:38:42.940438    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-53a4f4\"/\"kube-root-ca.crt\""
Jul 23 15:38:42.940778 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:38:42.940722    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-53a4f4\"/\"default-dockercfg-zpbqs\""
Jul 23 15:38:42.940778 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:38:42.940730    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-53a4f4\"/\"sub-test-model-53a4f4-kserve-self-signed-certs\""
Jul 23 15:38:42.943042 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:38:42.943019    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-53a4f4/sub-test-model-53a4f4-kserve-57dd967d5-dxbn9"]
Jul 23 15:38:43.019633 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:38:43.019590    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8063a980-3232-420b-8b65-fc17ce3d4e23-tmp-dir\") pod \"sub-test-model-53a4f4-kserve-57dd967d5-dxbn9\" (UID: \"8063a980-3232-420b-8b65-fc17ce3d4e23\") " pod="ai-tenant-e2e-shared-b-53a4f4/sub-test-model-53a4f4-kserve-57dd967d5-dxbn9"
Jul 23 15:38:43.019835 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:38:43.019649    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8063a980-3232-420b-8b65-fc17ce3d4e23-home\") pod \"sub-test-model-53a4f4-kserve-57dd967d5-dxbn9\" (UID: \"8063a980-3232-420b-8b65-fc17ce3d4e23\") " pod="ai-tenant-e2e-shared-b-53a4f4/sub-test-model-53a4f4-kserve-57dd967d5-dxbn9"
Jul 23 15:38:43.019835 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:38:43.019734    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8063a980-3232-420b-8b65-fc17ce3d4e23-dshm\") pod \"sub-test-model-53a4f4-kserve-57dd967d5-dxbn9\" (UID: \"8063a980-3232-420b-8b65-fc17ce3d4e23\") " pod="ai-tenant-e2e-shared-b-53a4f4/sub-test-model-53a4f4-kserve-57dd967d5-dxbn9"
Jul 23 15:38:43.019835 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:38:43.019771    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8063a980-3232-420b-8b65-fc17ce3d4e23-model-cache\") pod \"sub-test-model-53a4f4-kserve-57dd967d5-dxbn9\" (UID: \"8063a980-3232-420b-8b65-fc17ce3d4e23\") " pod="ai-tenant-e2e-shared-b-53a4f4/sub-test-model-53a4f4-kserve-57dd967d5-dxbn9"
Jul 23 15:38:43.019835 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:38:43.019813    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27ktr\" (UniqueName: \"kubernetes.io/projected/8063a980-3232-420b-8b65-fc17ce3d4e23-kube-api-access-27ktr\") pod \"sub-test-model-53a4f4-kserve-57dd967d5-dxbn9\" (UID: \"8063a980-3232-420b-8b65-fc17ce3d4e23\") " pod="ai-tenant-e2e-shared-b-53a4f4/sub-test-model-53a4f4-kserve-57dd967d5-dxbn9"
Jul 23 15:38:43.020029 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:38:43.019854    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8063a980-3232-420b-8b65-fc17ce3d4e23-tls-certs\") pod \"sub-test-model-53a4f4-kserve-57dd967d5-dxbn9\" (UID: \"8063a980-3232-420b-8b65-fc17ce3d4e23\") " pod="ai-tenant-e2e-shared-b-53a4f4/sub-test-model-53a4f4-kserve-57dd967d5-dxbn9"
Jul 23 15:38:43.121101 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:38:43.121069    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8063a980-3232-420b-8b65-fc17ce3d4e23-tls-certs\") pod \"sub-test-model-53a4f4-kserve-57dd967d5-dxbn9\" (UID: \"8063a980-3232-420b-8b65-fc17ce3d4e23\") " pod="ai-tenant-e2e-shared-b-53a4f4/sub-test-model-53a4f4-kserve-57dd967d5-dxbn9"
Jul 23 15:38:43.121320 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:38:43.121158    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8063a980-3232-420b-8b65-fc17ce3d4e23-tmp-dir\") pod \"sub-test-model-53a4f4-kserve-57dd967d5-dxbn9\" (UID: \"8063a980-3232-420b-8b65-fc17ce3d4e23\") " pod="ai-tenant-e2e-shared-b-53a4f4/sub-test-model-53a4f4-kserve-57dd967d5-dxbn9"
Jul 23 15:38:43.121320 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:38:43.121198    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8063a980-3232-420b-8b65-fc17ce3d4e23-home\") pod \"sub-test-model-53a4f4-kserve-57dd967d5-dxbn9\" (UID: \"8063a980-3232-420b-8b65-fc17ce3d4e23\") " pod="ai-tenant-e2e-shared-b-53a4f4/sub-test-model-53a4f4-kserve-57dd967d5-dxbn9"
Jul 23 15:38:43.121320 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:38:43.121225    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8063a980-3232-420b-8b65-fc17ce3d4e23-dshm\") pod \"sub-test-model-53a4f4-kserve-57dd967d5-dxbn9\" (UID: \"8063a980-3232-420b-8b65-fc17ce3d4e23\") " pod="ai-tenant-e2e-shared-b-53a4f4/sub-test-model-53a4f4-kserve-57dd967d5-dxbn9"
Jul 23 15:38:43.121320 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:38:43.121263    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8063a980-3232-420b-8b65-fc17ce3d4e23-model-cache\") pod \"sub-test-model-53a4f4-kserve-57dd967d5-dxbn9\" (UID: \"8063a980-3232-420b-8b65-fc17ce3d4e23\") " pod="ai-tenant-e2e-shared-b-53a4f4/sub-test-model-53a4f4-kserve-57dd967d5-dxbn9"
Jul 23 15:38:43.121507 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:38:43.121382    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27ktr\" (UniqueName: \"kubernetes.io/projected/8063a980-3232-420b-8b65-fc17ce3d4e23-kube-api-access-27ktr\") pod \"sub-test-model-53a4f4-kserve-57dd967d5-dxbn9\" (UID: \"8063a980-3232-420b-8b65-fc17ce3d4e23\") " pod="ai-tenant-e2e-shared-b-53a4f4/sub-test-model-53a4f4-kserve-57dd967d5-dxbn9"
Jul 23 15:38:43.121629 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:38:43.121606    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8063a980-3232-420b-8b65-fc17ce3d4e23-model-cache\") pod \"sub-test-model-53a4f4-kserve-57dd967d5-dxbn9\" (UID: \"8063a980-3232-420b-8b65-fc17ce3d4e23\") " pod="ai-tenant-e2e-shared-b-53a4f4/sub-test-model-53a4f4-kserve-57dd967d5-dxbn9"
Jul 23 15:38:43.121629 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:38:43.121619    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8063a980-3232-420b-8b65-fc17ce3d4e23-home\") pod \"sub-test-model-53a4f4-kserve-57dd967d5-dxbn9\" (UID: \"8063a980-3232-420b-8b65-fc17ce3d4e23\") " pod="ai-tenant-e2e-shared-b-53a4f4/sub-test-model-53a4f4-kserve-57dd967d5-dxbn9"
Jul 23 15:38:43.121741 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:38:43.121684    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8063a980-3232-420b-8b65-fc17ce3d4e23-tmp-dir\") pod \"sub-test-model-53a4f4-kserve-57dd967d5-dxbn9\" (UID: \"8063a980-3232-420b-8b65-fc17ce3d4e23\") " pod="ai-tenant-e2e-shared-b-53a4f4/sub-test-model-53a4f4-kserve-57dd967d5-dxbn9"
Jul 23 15:38:43.123528 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:38:43.123507    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8063a980-3232-420b-8b65-fc17ce3d4e23-dshm\") pod \"sub-test-model-53a4f4-kserve-57dd967d5-dxbn9\" (UID: \"8063a980-3232-420b-8b65-fc17ce3d4e23\") " pod="ai-tenant-e2e-shared-b-53a4f4/sub-test-model-53a4f4-kserve-57dd967d5-dxbn9"
Jul 23 15:38:43.123840 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:38:43.123824    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8063a980-3232-420b-8b65-fc17ce3d4e23-tls-certs\") pod \"sub-test-model-53a4f4-kserve-57dd967d5-dxbn9\" (UID: \"8063a980-3232-420b-8b65-fc17ce3d4e23\") " pod="ai-tenant-e2e-shared-b-53a4f4/sub-test-model-53a4f4-kserve-57dd967d5-dxbn9"
Jul 23 15:38:43.129887 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:38:43.129856    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-27ktr\" (UniqueName: \"kubernetes.io/projected/8063a980-3232-420b-8b65-fc17ce3d4e23-kube-api-access-27ktr\") pod \"sub-test-model-53a4f4-kserve-57dd967d5-dxbn9\" (UID: \"8063a980-3232-420b-8b65-fc17ce3d4e23\") " pod="ai-tenant-e2e-shared-b-53a4f4/sub-test-model-53a4f4-kserve-57dd967d5-dxbn9"
Jul 23 15:38:43.246463 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:38:43.246433    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-53a4f4/sub-test-model-53a4f4-kserve-57dd967d5-dxbn9"
Jul 23 15:38:43.385435 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:38:43.385410    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-53a4f4/sub-test-model-53a4f4-kserve-57dd967d5-dxbn9"]
Jul 23 15:38:43.387970 ip-10-0-141-105 kubenswrapper[2560]: W0723 15:38:43.387939    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8063a980_3232_420b_8b65_fc17ce3d4e23.slice/crio-4f176e48b1047e4c9c4ff42f4bdb70689c329897b6430523978f9ad36ffe161d WatchSource:0}: Error finding container 4f176e48b1047e4c9c4ff42f4bdb70689c329897b6430523978f9ad36ffe161d: Status 404 returned error can't find the container with id 4f176e48b1047e4c9c4ff42f4bdb70689c329897b6430523978f9ad36ffe161d
Jul 23 15:38:43.498680 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:38:43.498600    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-53a4f4/sub-test-model-53a4f4-kserve-57dd967d5-dxbn9" event={"ID":"8063a980-3232-420b-8b65-fc17ce3d4e23","Type":"ContainerStarted","Data":"53b5abe17525f608594d960075b053cea4660f08237621a3b2a394a122c3a0cc"}
Jul 23 15:38:43.498680 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:38:43.498644    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-53a4f4/sub-test-model-53a4f4-kserve-57dd967d5-dxbn9" event={"ID":"8063a980-3232-420b-8b65-fc17ce3d4e23","Type":"ContainerStarted","Data":"4f176e48b1047e4c9c4ff42f4bdb70689c329897b6430523978f9ad36ffe161d"}
Jul 23 15:38:43.517455 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:38:43.517403    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-53a4f4/sub-test-model-53a4f4-kserve-57dd967d5-dxbn9" podStartSLOduration=1.5173881580000002 podStartE2EDuration="1.517388158s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 15:38:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 15:38:43.514672653 +0000 UTC m=+2765.022546401" watchObservedRunningTime="2026-07-23 15:38:43.517388158 +0000 UTC m=+2765.025261954"
Jul 23 15:38:47.022826 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:38:47.022787    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cgwsg"]
Jul 23 15:38:53.247424 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:38:53.247385    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-53a4f4/sub-test-model-53a4f4-kserve-57dd967d5-dxbn9"
Jul 23 15:38:53.247424 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:38:53.247433    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-53a4f4/sub-test-model-53a4f4-kserve-57dd967d5-dxbn9"
Jul 23 15:38:53.259802 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:38:53.259775    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-53a4f4/sub-test-model-53a4f4-kserve-57dd967d5-dxbn9"
Jul 23 15:38:53.556644 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:38:53.556556    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-53a4f4/sub-test-model-53a4f4-kserve-57dd967d5-dxbn9"
Jul 23 15:39:01.533019 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:39:01.532930    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cgwsg"]
Jul 23 15:39:20.031673 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:39:20.031632    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cgwsg"]
Jul 23 15:39:28.665874 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:39:28.665837    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-53a4f4/sub-test-model-53a4f4-kserve-57dd967d5-dxbn9"]
Jul 23 15:39:39.327292 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:39:39.327253    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cgwsg"]
Jul 23 15:39:43.686416 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:39:43.686373    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-53a4f4/sub-test-model-53a4f4-kserve-57dd967d5-dxbn9" podUID="8063a980-3232-420b-8b65-fc17ce3d4e23" containerName="main" containerID="cri-o://53b5abe17525f608594d960075b053cea4660f08237621a3b2a394a122c3a0cc" gracePeriod=45
Jul 23 15:39:43.947794 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:39:43.947728    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-53a4f4/sub-test-model-53a4f4-kserve-57dd967d5-dxbn9"
Jul 23 15:39:43.978017 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:39:43.977986    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8063a980-3232-420b-8b65-fc17ce3d4e23-dshm\") pod \"8063a980-3232-420b-8b65-fc17ce3d4e23\" (UID: \"8063a980-3232-420b-8b65-fc17ce3d4e23\") "
Jul 23 15:39:43.978218 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:39:43.978027    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8063a980-3232-420b-8b65-fc17ce3d4e23-tmp-dir\") pod \"8063a980-3232-420b-8b65-fc17ce3d4e23\" (UID: \"8063a980-3232-420b-8b65-fc17ce3d4e23\") "
Jul 23 15:39:43.978218 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:39:43.978046    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-27ktr\" (UniqueName: \"kubernetes.io/projected/8063a980-3232-420b-8b65-fc17ce3d4e23-kube-api-access-27ktr\") pod \"8063a980-3232-420b-8b65-fc17ce3d4e23\" (UID: \"8063a980-3232-420b-8b65-fc17ce3d4e23\") "
Jul 23 15:39:43.978218 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:39:43.978111    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8063a980-3232-420b-8b65-fc17ce3d4e23-model-cache\") pod \"8063a980-3232-420b-8b65-fc17ce3d4e23\" (UID: \"8063a980-3232-420b-8b65-fc17ce3d4e23\") "
Jul 23 15:39:43.978218 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:39:43.978145    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8063a980-3232-420b-8b65-fc17ce3d4e23-tls-certs\") pod \"8063a980-3232-420b-8b65-fc17ce3d4e23\" (UID: \"8063a980-3232-420b-8b65-fc17ce3d4e23\") "
Jul 23 15:39:43.978218 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:39:43.978183    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8063a980-3232-420b-8b65-fc17ce3d4e23-home\") pod \"8063a980-3232-420b-8b65-fc17ce3d4e23\" (UID: \"8063a980-3232-420b-8b65-fc17ce3d4e23\") "
Jul 23 15:39:43.978522 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:39:43.978351    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8063a980-3232-420b-8b65-fc17ce3d4e23-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8063a980-3232-420b-8b65-fc17ce3d4e23" (UID: "8063a980-3232-420b-8b65-fc17ce3d4e23"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 15:39:43.978522 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:39:43.978418    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8063a980-3232-420b-8b65-fc17ce3d4e23-model-cache" (OuterVolumeSpecName: "model-cache") pod "8063a980-3232-420b-8b65-fc17ce3d4e23" (UID: "8063a980-3232-420b-8b65-fc17ce3d4e23"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 15:39:43.978522 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:39:43.978503    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8063a980-3232-420b-8b65-fc17ce3d4e23-tmp-dir\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Jul 23 15:39:43.978661 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:39:43.978524    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8063a980-3232-420b-8b65-fc17ce3d4e23-model-cache\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Jul 23 15:39:43.978661 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:39:43.978554    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8063a980-3232-420b-8b65-fc17ce3d4e23-home" (OuterVolumeSpecName: "home") pod "8063a980-3232-420b-8b65-fc17ce3d4e23" (UID: "8063a980-3232-420b-8b65-fc17ce3d4e23"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 15:39:43.980573 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:39:43.980539    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8063a980-3232-420b-8b65-fc17ce3d4e23-dshm" (OuterVolumeSpecName: "dshm") pod "8063a980-3232-420b-8b65-fc17ce3d4e23" (UID: "8063a980-3232-420b-8b65-fc17ce3d4e23"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 15:39:43.980702 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:39:43.980596    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8063a980-3232-420b-8b65-fc17ce3d4e23-kube-api-access-27ktr" (OuterVolumeSpecName: "kube-api-access-27ktr") pod "8063a980-3232-420b-8b65-fc17ce3d4e23" (UID: "8063a980-3232-420b-8b65-fc17ce3d4e23"). InnerVolumeSpecName "kube-api-access-27ktr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 15:39:43.980702 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:39:43.980600    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8063a980-3232-420b-8b65-fc17ce3d4e23-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8063a980-3232-420b-8b65-fc17ce3d4e23" (UID: "8063a980-3232-420b-8b65-fc17ce3d4e23"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 15:39:44.025886 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:39:44.025849    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-d3e0e7/rate-test-model-d3e0e7-kserve-8594b9d46c-rzh8k"]
Jul 23 15:39:44.026247 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:39:44.026218    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8063a980-3232-420b-8b65-fc17ce3d4e23" containerName="main"
Jul 23 15:39:44.026382 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:39:44.026255    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="8063a980-3232-420b-8b65-fc17ce3d4e23" containerName="main"
Jul 23 15:39:44.026382 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:39:44.026314    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="8063a980-3232-420b-8b65-fc17ce3d4e23" containerName="main"
Jul 23 15:39:44.029421 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:39:44.029398    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-d3e0e7/rate-test-model-d3e0e7-kserve-8594b9d46c-rzh8k"
Jul 23 15:39:44.032102 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:39:44.032076    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-d3e0e7\"/\"kube-root-ca.crt\""
Jul 23 15:39:44.032102 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:39:44.032091    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-d3e0e7\"/\"rate-test-model-d3e0e7-kserve-self-signed-certs\""
Jul 23 15:39:44.032317 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:39:44.032130    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-d3e0e7\"/\"openshift-service-ca.crt\""
Jul 23 15:39:44.032317 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:39:44.032135    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-d3e0e7\"/\"default-dockercfg-5h5kt\""
Jul 23 15:39:44.041110 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:39:44.041081    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-d3e0e7/rate-test-model-d3e0e7-kserve-8594b9d46c-rzh8k"]
Jul 23 15:39:44.079944 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:39:44.079903    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/848d6291-4db9-44d5-a57a-1b12c9a3d848-model-cache\") pod \"rate-test-model-d3e0e7-kserve-8594b9d46c-rzh8k\" (UID: \"848d6291-4db9-44d5-a57a-1b12c9a3d848\") " pod="ai-tenant-e2e-shared-a-d3e0e7/rate-test-model-d3e0e7-kserve-8594b9d46c-rzh8k"
Jul 23 15:39:44.080138 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:39:44.079955    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/848d6291-4db9-44d5-a57a-1b12c9a3d848-tls-certs\") pod \"rate-test-model-d3e0e7-kserve-8594b9d46c-rzh8k\" (UID: \"848d6291-4db9-44d5-a57a-1b12c9a3d848\") " pod="ai-tenant-e2e-shared-a-d3e0e7/rate-test-model-d3e0e7-kserve-8594b9d46c-rzh8k"
Jul 23 15:39:44.080138 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:39:44.080007    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/848d6291-4db9-44d5-a57a-1b12c9a3d848-tmp-dir\") pod \"rate-test-model-d3e0e7-kserve-8594b9d46c-rzh8k\" (UID: \"848d6291-4db9-44d5-a57a-1b12c9a3d848\") " pod="ai-tenant-e2e-shared-a-d3e0e7/rate-test-model-d3e0e7-kserve-8594b9d46c-rzh8k"
Jul 23 15:39:44.080138 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:39:44.080031    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzn4z\" (UniqueName: \"kubernetes.io/projected/848d6291-4db9-44d5-a57a-1b12c9a3d848-kube-api-access-mzn4z\") pod \"rate-test-model-d3e0e7-kserve-8594b9d46c-rzh8k\" (UID: \"848d6291-4db9-44d5-a57a-1b12c9a3d848\") " pod="ai-tenant-e2e-shared-a-d3e0e7/rate-test-model-d3e0e7-kserve-8594b9d46c-rzh8k"
Jul 23 15:39:44.080320 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:39:44.080134    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/848d6291-4db9-44d5-a57a-1b12c9a3d848-home\") pod \"rate-test-model-d3e0e7-kserve-8594b9d46c-rzh8k\" (UID: \"848d6291-4db9-44d5-a57a-1b12c9a3d848\") " pod="ai-tenant-e2e-shared-a-d3e0e7/rate-test-model-d3e0e7-kserve-8594b9d46c-rzh8k"
Jul 23 15:39:44.080320 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:39:44.080167    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/848d6291-4db9-44d5-a57a-1b12c9a3d848-dshm\") pod \"rate-test-model-d3e0e7-kserve-8594b9d46c-rzh8k\" (UID: \"848d6291-4db9-44d5-a57a-1b12c9a3d848\") " pod="ai-tenant-e2e-shared-a-d3e0e7/rate-test-model-d3e0e7-kserve-8594b9d46c-rzh8k"
Jul 23 15:39:44.080320 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:39:44.080267    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8063a980-3232-420b-8b65-fc17ce3d4e23-dshm\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Jul 23 15:39:44.080320 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:39:44.080278    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27ktr\" (UniqueName: \"kubernetes.io/projected/8063a980-3232-420b-8b65-fc17ce3d4e23-kube-api-access-27ktr\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Jul 23 15:39:44.080320 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:39:44.080289    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8063a980-3232-420b-8b65-fc17ce3d4e23-tls-certs\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Jul 23 15:39:44.080320 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:39:44.080298    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8063a980-3232-420b-8b65-fc17ce3d4e23-home\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Jul 23 15:39:44.181112 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:39:44.181071    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/848d6291-4db9-44d5-a57a-1b12c9a3d848-model-cache\") pod \"rate-test-model-d3e0e7-kserve-8594b9d46c-rzh8k\" (UID: \"848d6291-4db9-44d5-a57a-1b12c9a3d848\") " pod="ai-tenant-e2e-shared-a-d3e0e7/rate-test-model-d3e0e7-kserve-8594b9d46c-rzh8k"
Jul 23 15:39:44.181112 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:39:44.181117    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/848d6291-4db9-44d5-a57a-1b12c9a3d848-tls-certs\") pod \"rate-test-model-d3e0e7-kserve-8594b9d46c-rzh8k\" (UID: \"848d6291-4db9-44d5-a57a-1b12c9a3d848\") " pod="ai-tenant-e2e-shared-a-d3e0e7/rate-test-model-d3e0e7-kserve-8594b9d46c-rzh8k"
Jul 23 15:39:44.181435 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:39:44.181298    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/848d6291-4db9-44d5-a57a-1b12c9a3d848-tmp-dir\") pod \"rate-test-model-d3e0e7-kserve-8594b9d46c-rzh8k\" (UID: \"848d6291-4db9-44d5-a57a-1b12c9a3d848\") " pod="ai-tenant-e2e-shared-a-d3e0e7/rate-test-model-d3e0e7-kserve-8594b9d46c-rzh8k"
Jul 23 15:39:44.181435 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:39:44.181329    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzn4z\" (UniqueName: \"kubernetes.io/projected/848d6291-4db9-44d5-a57a-1b12c9a3d848-kube-api-access-mzn4z\") pod \"rate-test-model-d3e0e7-kserve-8594b9d46c-rzh8k\" (UID: \"848d6291-4db9-44d5-a57a-1b12c9a3d848\") " pod="ai-tenant-e2e-shared-a-d3e0e7/rate-test-model-d3e0e7-kserve-8594b9d46c-rzh8k"
Jul 23 15:39:44.181556 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:39:44.181442    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/848d6291-4db9-44d5-a57a-1b12c9a3d848-home\") pod \"rate-test-model-d3e0e7-kserve-8594b9d46c-rzh8k\" (UID: \"848d6291-4db9-44d5-a57a-1b12c9a3d848\") " pod="ai-tenant-e2e-shared-a-d3e0e7/rate-test-model-d3e0e7-kserve-8594b9d46c-rzh8k"
Jul 23 15:39:44.181556 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:39:44.181480    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/848d6291-4db9-44d5-a57a-1b12c9a3d848-dshm\") pod \"rate-test-model-d3e0e7-kserve-8594b9d46c-rzh8k\" (UID: \"848d6291-4db9-44d5-a57a-1b12c9a3d848\") " pod="ai-tenant-e2e-shared-a-d3e0e7/rate-test-model-d3e0e7-kserve-8594b9d46c-rzh8k"
Jul 23 15:39:44.181669 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:39:44.181557    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/848d6291-4db9-44d5-a57a-1b12c9a3d848-model-cache\") pod \"rate-test-model-d3e0e7-kserve-8594b9d46c-rzh8k\" (UID: \"848d6291-4db9-44d5-a57a-1b12c9a3d848\") " pod="ai-tenant-e2e-shared-a-d3e0e7/rate-test-model-d3e0e7-kserve-8594b9d46c-rzh8k"
Jul 23 15:39:44.181669 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:39:44.181604    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/848d6291-4db9-44d5-a57a-1b12c9a3d848-tmp-dir\") pod \"rate-test-model-d3e0e7-kserve-8594b9d46c-rzh8k\" (UID: \"848d6291-4db9-44d5-a57a-1b12c9a3d848\") " pod="ai-tenant-e2e-shared-a-d3e0e7/rate-test-model-d3e0e7-kserve-8594b9d46c-rzh8k"
Jul 23 15:39:44.181806 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:39:44.181719    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/848d6291-4db9-44d5-a57a-1b12c9a3d848-home\") pod \"rate-test-model-d3e0e7-kserve-8594b9d46c-rzh8k\" (UID: \"848d6291-4db9-44d5-a57a-1b12c9a3d848\") " pod="ai-tenant-e2e-shared-a-d3e0e7/rate-test-model-d3e0e7-kserve-8594b9d46c-rzh8k"
Jul 23 15:39:44.183696 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:39:44.183675    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/848d6291-4db9-44d5-a57a-1b12c9a3d848-dshm\") pod \"rate-test-model-d3e0e7-kserve-8594b9d46c-rzh8k\" (UID: \"848d6291-4db9-44d5-a57a-1b12c9a3d848\") " pod="ai-tenant-e2e-shared-a-d3e0e7/rate-test-model-d3e0e7-kserve-8594b9d46c-rzh8k"
Jul 23 15:39:44.183783 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:39:44.183730    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/848d6291-4db9-44d5-a57a-1b12c9a3d848-tls-certs\") pod \"rate-test-model-d3e0e7-kserve-8594b9d46c-rzh8k\" (UID: \"848d6291-4db9-44d5-a57a-1b12c9a3d848\") " pod="ai-tenant-e2e-shared-a-d3e0e7/rate-test-model-d3e0e7-kserve-8594b9d46c-rzh8k"
Jul 23 15:39:44.190331 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:39:44.190314    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzn4z\" (UniqueName: \"kubernetes.io/projected/848d6291-4db9-44d5-a57a-1b12c9a3d848-kube-api-access-mzn4z\") pod \"rate-test-model-d3e0e7-kserve-8594b9d46c-rzh8k\" (UID: \"848d6291-4db9-44d5-a57a-1b12c9a3d848\") " pod="ai-tenant-e2e-shared-a-d3e0e7/rate-test-model-d3e0e7-kserve-8594b9d46c-rzh8k"
Jul 23 15:39:44.340296 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:39:44.340259    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-d3e0e7/rate-test-model-d3e0e7-kserve-8594b9d46c-rzh8k"
Jul 23 15:39:44.477524 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:39:44.477496    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-d3e0e7/rate-test-model-d3e0e7-kserve-8594b9d46c-rzh8k"]
Jul 23 15:39:44.480157 ip-10-0-141-105 kubenswrapper[2560]: W0723 15:39:44.480131    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod848d6291_4db9_44d5_a57a_1b12c9a3d848.slice/crio-72d08cd4553b8e862829ea33e6f6d7e3e301bd7f5375f60b74433cf35e89f0bf WatchSource:0}: Error finding container 72d08cd4553b8e862829ea33e6f6d7e3e301bd7f5375f60b74433cf35e89f0bf: Status 404 returned error can't find the container with id 72d08cd4553b8e862829ea33e6f6d7e3e301bd7f5375f60b74433cf35e89f0bf
Jul 23 15:39:44.794848 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:39:44.794811    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-d3e0e7/rate-test-model-d3e0e7-kserve-8594b9d46c-rzh8k" event={"ID":"848d6291-4db9-44d5-a57a-1b12c9a3d848","Type":"ContainerStarted","Data":"53e5680acceeff6cb6b844dfd04809f34bf5b06ca4e3f67356be785cc435978e"}
Jul 23 15:39:44.795349 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:39:44.794860    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-d3e0e7/rate-test-model-d3e0e7-kserve-8594b9d46c-rzh8k" event={"ID":"848d6291-4db9-44d5-a57a-1b12c9a3d848","Type":"ContainerStarted","Data":"72d08cd4553b8e862829ea33e6f6d7e3e301bd7f5375f60b74433cf35e89f0bf"}
Jul 23 15:39:44.796132 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:39:44.796109    2560 generic.go:356] "Generic (PLEG): container finished" podID="8063a980-3232-420b-8b65-fc17ce3d4e23" containerID="53b5abe17525f608594d960075b053cea4660f08237621a3b2a394a122c3a0cc" exitCode=0
Jul 23 15:39:44.796183 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:39:44.796143    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-53a4f4/sub-test-model-53a4f4-kserve-57dd967d5-dxbn9" event={"ID":"8063a980-3232-420b-8b65-fc17ce3d4e23","Type":"ContainerDied","Data":"53b5abe17525f608594d960075b053cea4660f08237621a3b2a394a122c3a0cc"}
Jul 23 15:39:44.796183 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:39:44.796148    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-53a4f4/sub-test-model-53a4f4-kserve-57dd967d5-dxbn9"
Jul 23 15:39:44.796183 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:39:44.796167    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-53a4f4/sub-test-model-53a4f4-kserve-57dd967d5-dxbn9" event={"ID":"8063a980-3232-420b-8b65-fc17ce3d4e23","Type":"ContainerDied","Data":"4f176e48b1047e4c9c4ff42f4bdb70689c329897b6430523978f9ad36ffe161d"}
Jul 23 15:39:44.796305 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:39:44.796184    2560 scope.go:117] "RemoveContainer" containerID="53b5abe17525f608594d960075b053cea4660f08237621a3b2a394a122c3a0cc"
Jul 23 15:39:44.805123 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:39:44.805103    2560 scope.go:117] "RemoveContainer" containerID="53b5abe17525f608594d960075b053cea4660f08237621a3b2a394a122c3a0cc"
Jul 23 15:39:44.805431 ip-10-0-141-105 kubenswrapper[2560]: E0723 15:39:44.805407    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53b5abe17525f608594d960075b053cea4660f08237621a3b2a394a122c3a0cc\": container with ID starting with 53b5abe17525f608594d960075b053cea4660f08237621a3b2a394a122c3a0cc not found: ID does not exist" containerID="53b5abe17525f608594d960075b053cea4660f08237621a3b2a394a122c3a0cc"
Jul 23 15:39:44.805515 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:39:44.805439    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53b5abe17525f608594d960075b053cea4660f08237621a3b2a394a122c3a0cc"} err="failed to get container status \"53b5abe17525f608594d960075b053cea4660f08237621a3b2a394a122c3a0cc\": rpc error: code = NotFound desc = could not find container \"53b5abe17525f608594d960075b053cea4660f08237621a3b2a394a122c3a0cc\": container with ID starting with 53b5abe17525f608594d960075b053cea4660f08237621a3b2a394a122c3a0cc not found: ID does not exist"
Jul 23 15:39:44.811065 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:39:44.811025    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-d3e0e7/rate-test-model-d3e0e7-kserve-8594b9d46c-rzh8k" podStartSLOduration=0.811012174 podStartE2EDuration="811.012174ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 15:39:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 15:39:44.810630097 +0000 UTC m=+2826.318503845" watchObservedRunningTime="2026-07-23 15:39:44.811012174 +0000 UTC m=+2826.318885921"
Jul 23 15:39:44.822587 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:39:44.822555    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-53a4f4/sub-test-model-53a4f4-kserve-57dd967d5-dxbn9"]
Jul 23 15:39:44.825934 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:39:44.825912    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-53a4f4/sub-test-model-53a4f4-kserve-57dd967d5-dxbn9"]
Jul 23 15:39:45.063793 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:39:45.063708    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8063a980-3232-420b-8b65-fc17ce3d4e23" path="/var/lib/kubelet/pods/8063a980-3232-420b-8b65-fc17ce3d4e23/volumes"
Jul 23 15:39:49.022936 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:39:49.022904    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cgwsg"]
Jul 23 15:39:54.341025 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:39:54.340983    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-d3e0e7/rate-test-model-d3e0e7-kserve-8594b9d46c-rzh8k"
Jul 23 15:39:54.341575 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:39:54.341041    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-d3e0e7/rate-test-model-d3e0e7-kserve-8594b9d46c-rzh8k"
Jul 23 15:39:54.353902 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:39:54.353875    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-d3e0e7/rate-test-model-d3e0e7-kserve-8594b9d46c-rzh8k"
Jul 23 15:39:54.857980 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:39:54.857951    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-d3e0e7/rate-test-model-d3e0e7-kserve-8594b9d46c-rzh8k"
Jul 23 15:40:06.833186 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:40:06.833153    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-53a4f4/rate-test-model-53a4f4-kserve-7d76959979-pszlt"]
Jul 23 15:40:06.837151 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:40:06.837135    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-53a4f4/rate-test-model-53a4f4-kserve-7d76959979-pszlt"
Jul 23 15:40:06.839877 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:40:06.839852    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-53a4f4\"/\"openshift-service-ca.crt\""
Jul 23 15:40:06.840021 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:40:06.839853    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-53a4f4\"/\"kube-root-ca.crt\""
Jul 23 15:40:06.841069 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:40:06.841045    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-53a4f4\"/\"default-dockercfg-zpbqs\""
Jul 23 15:40:06.841202 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:40:06.841118    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-53a4f4\"/\"rate-test-model-53a4f4-kserve-self-signed-certs\""
Jul 23 15:40:06.843982 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:40:06.843958    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-53a4f4/rate-test-model-53a4f4-kserve-7d76959979-pszlt"]
Jul 23 15:40:06.992734 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:40:06.992679    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c81c1c6f-8867-41e7-b559-0adecfb7049e-tmp-dir\") pod \"rate-test-model-53a4f4-kserve-7d76959979-pszlt\" (UID: \"c81c1c6f-8867-41e7-b559-0adecfb7049e\") " pod="ai-tenant-e2e-shared-b-53a4f4/rate-test-model-53a4f4-kserve-7d76959979-pszlt"
Jul 23 15:40:06.992734 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:40:06.992733    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c81c1c6f-8867-41e7-b559-0adecfb7049e-model-cache\") pod \"rate-test-model-53a4f4-kserve-7d76959979-pszlt\" (UID: \"c81c1c6f-8867-41e7-b559-0adecfb7049e\") " pod="ai-tenant-e2e-shared-b-53a4f4/rate-test-model-53a4f4-kserve-7d76959979-pszlt"
Jul 23 15:40:06.993031 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:40:06.992765    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c81c1c6f-8867-41e7-b559-0adecfb7049e-dshm\") pod \"rate-test-model-53a4f4-kserve-7d76959979-pszlt\" (UID: \"c81c1c6f-8867-41e7-b559-0adecfb7049e\") " pod="ai-tenant-e2e-shared-b-53a4f4/rate-test-model-53a4f4-kserve-7d76959979-pszlt"
Jul 23 15:40:06.993031 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:40:06.992862    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c81c1c6f-8867-41e7-b559-0adecfb7049e-home\") pod \"rate-test-model-53a4f4-kserve-7d76959979-pszlt\" (UID: \"c81c1c6f-8867-41e7-b559-0adecfb7049e\") " pod="ai-tenant-e2e-shared-b-53a4f4/rate-test-model-53a4f4-kserve-7d76959979-pszlt"
Jul 23 15:40:06.993031 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:40:06.992913    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4p26\" (UniqueName: \"kubernetes.io/projected/c81c1c6f-8867-41e7-b559-0adecfb7049e-kube-api-access-z4p26\") pod \"rate-test-model-53a4f4-kserve-7d76959979-pszlt\" (UID: \"c81c1c6f-8867-41e7-b559-0adecfb7049e\") " pod="ai-tenant-e2e-shared-b-53a4f4/rate-test-model-53a4f4-kserve-7d76959979-pszlt"
Jul 23 15:40:06.993031 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:40:06.993011    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c81c1c6f-8867-41e7-b559-0adecfb7049e-tls-certs\") pod \"rate-test-model-53a4f4-kserve-7d76959979-pszlt\" (UID: \"c81c1c6f-8867-41e7-b559-0adecfb7049e\") " pod="ai-tenant-e2e-shared-b-53a4f4/rate-test-model-53a4f4-kserve-7d76959979-pszlt"
Jul 23 15:40:07.094595 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:40:07.094488    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4p26\" (UniqueName: \"kubernetes.io/projected/c81c1c6f-8867-41e7-b559-0adecfb7049e-kube-api-access-z4p26\") pod \"rate-test-model-53a4f4-kserve-7d76959979-pszlt\" (UID: \"c81c1c6f-8867-41e7-b559-0adecfb7049e\") " pod="ai-tenant-e2e-shared-b-53a4f4/rate-test-model-53a4f4-kserve-7d76959979-pszlt"
Jul 23 15:40:07.094595 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:40:07.094585    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c81c1c6f-8867-41e7-b559-0adecfb7049e-tls-certs\") pod \"rate-test-model-53a4f4-kserve-7d76959979-pszlt\" (UID: \"c81c1c6f-8867-41e7-b559-0adecfb7049e\") " pod="ai-tenant-e2e-shared-b-53a4f4/rate-test-model-53a4f4-kserve-7d76959979-pszlt"
Jul 23 15:40:07.094840 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:40:07.094647    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c81c1c6f-8867-41e7-b559-0adecfb7049e-tmp-dir\") pod \"rate-test-model-53a4f4-kserve-7d76959979-pszlt\" (UID: \"c81c1c6f-8867-41e7-b559-0adecfb7049e\") " pod="ai-tenant-e2e-shared-b-53a4f4/rate-test-model-53a4f4-kserve-7d76959979-pszlt"
Jul 23 15:40:07.094840 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:40:07.094677    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c81c1c6f-8867-41e7-b559-0adecfb7049e-model-cache\") pod \"rate-test-model-53a4f4-kserve-7d76959979-pszlt\" (UID: \"c81c1c6f-8867-41e7-b559-0adecfb7049e\") " pod="ai-tenant-e2e-shared-b-53a4f4/rate-test-model-53a4f4-kserve-7d76959979-pszlt"
Jul 23 15:40:07.094840 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:40:07.094705    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c81c1c6f-8867-41e7-b559-0adecfb7049e-dshm\") pod \"rate-test-model-53a4f4-kserve-7d76959979-pszlt\" (UID: \"c81c1c6f-8867-41e7-b559-0adecfb7049e\") " pod="ai-tenant-e2e-shared-b-53a4f4/rate-test-model-53a4f4-kserve-7d76959979-pszlt"
Jul 23 15:40:07.094840 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:40:07.094750    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c81c1c6f-8867-41e7-b559-0adecfb7049e-home\") pod \"rate-test-model-53a4f4-kserve-7d76959979-pszlt\" (UID: \"c81c1c6f-8867-41e7-b559-0adecfb7049e\") " pod="ai-tenant-e2e-shared-b-53a4f4/rate-test-model-53a4f4-kserve-7d76959979-pszlt"
Jul 23 15:40:07.095179 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:40:07.095147    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c81c1c6f-8867-41e7-b559-0adecfb7049e-home\") pod \"rate-test-model-53a4f4-kserve-7d76959979-pszlt\" (UID: \"c81c1c6f-8867-41e7-b559-0adecfb7049e\") " pod="ai-tenant-e2e-shared-b-53a4f4/rate-test-model-53a4f4-kserve-7d76959979-pszlt"
Jul 23 15:40:07.095323 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:40:07.095214    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c81c1c6f-8867-41e7-b559-0adecfb7049e-model-cache\") pod \"rate-test-model-53a4f4-kserve-7d76959979-pszlt\" (UID: \"c81c1c6f-8867-41e7-b559-0adecfb7049e\") " pod="ai-tenant-e2e-shared-b-53a4f4/rate-test-model-53a4f4-kserve-7d76959979-pszlt"
Jul 23 15:40:07.095323 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:40:07.095262    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c81c1c6f-8867-41e7-b559-0adecfb7049e-tmp-dir\") pod \"rate-test-model-53a4f4-kserve-7d76959979-pszlt\" (UID: \"c81c1c6f-8867-41e7-b559-0adecfb7049e\") " pod="ai-tenant-e2e-shared-b-53a4f4/rate-test-model-53a4f4-kserve-7d76959979-pszlt"
Jul 23 15:40:07.097244 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:40:07.097208    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c81c1c6f-8867-41e7-b559-0adecfb7049e-dshm\") pod \"rate-test-model-53a4f4-kserve-7d76959979-pszlt\" (UID: \"c81c1c6f-8867-41e7-b559-0adecfb7049e\") " pod="ai-tenant-e2e-shared-b-53a4f4/rate-test-model-53a4f4-kserve-7d76959979-pszlt"
Jul 23 15:40:07.097507 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:40:07.097487    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c81c1c6f-8867-41e7-b559-0adecfb7049e-tls-certs\") pod \"rate-test-model-53a4f4-kserve-7d76959979-pszlt\" (UID: \"c81c1c6f-8867-41e7-b559-0adecfb7049e\") " pod="ai-tenant-e2e-shared-b-53a4f4/rate-test-model-53a4f4-kserve-7d76959979-pszlt"
Jul 23 15:40:07.102611 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:40:07.102588    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4p26\" (UniqueName: \"kubernetes.io/projected/c81c1c6f-8867-41e7-b559-0adecfb7049e-kube-api-access-z4p26\") pod \"rate-test-model-53a4f4-kserve-7d76959979-pszlt\" (UID: \"c81c1c6f-8867-41e7-b559-0adecfb7049e\") " pod="ai-tenant-e2e-shared-b-53a4f4/rate-test-model-53a4f4-kserve-7d76959979-pszlt"
Jul 23 15:40:07.148652 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:40:07.148599    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-53a4f4/rate-test-model-53a4f4-kserve-7d76959979-pszlt"
Jul 23 15:40:07.295124 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:40:07.294917    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-53a4f4/rate-test-model-53a4f4-kserve-7d76959979-pszlt"]
Jul 23 15:40:07.297927 ip-10-0-141-105 kubenswrapper[2560]: W0723 15:40:07.297900    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc81c1c6f_8867_41e7_b559_0adecfb7049e.slice/crio-cc801f56cc1d67619cac85bc2d48febf427eb9962259de370689f4a0aec33f95 WatchSource:0}: Error finding container cc801f56cc1d67619cac85bc2d48febf427eb9962259de370689f4a0aec33f95: Status 404 returned error can't find the container with id cc801f56cc1d67619cac85bc2d48febf427eb9962259de370689f4a0aec33f95
Jul 23 15:40:07.909944 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:40:07.909907    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-53a4f4/rate-test-model-53a4f4-kserve-7d76959979-pszlt" event={"ID":"c81c1c6f-8867-41e7-b559-0adecfb7049e","Type":"ContainerStarted","Data":"8d27eb68cbe51392960a48989c52f828dc07b5caf7da1b2661e6c3e404fd68c1"}
Jul 23 15:40:07.909944 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:40:07.909950    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-53a4f4/rate-test-model-53a4f4-kserve-7d76959979-pszlt" event={"ID":"c81c1c6f-8867-41e7-b559-0adecfb7049e","Type":"ContainerStarted","Data":"cc801f56cc1d67619cac85bc2d48febf427eb9962259de370689f4a0aec33f95"}
Jul 23 15:40:07.927277 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:40:07.927211    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-53a4f4/rate-test-model-53a4f4-kserve-7d76959979-pszlt" podStartSLOduration=1.927195558 podStartE2EDuration="1.927195558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 15:40:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 15:40:07.925428547 +0000 UTC m=+2849.433302296" watchObservedRunningTime="2026-07-23 15:40:07.927195558 +0000 UTC m=+2849.435069304"
Jul 23 15:40:09.423875 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:40:09.423839    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cgwsg"]
Jul 23 15:40:17.149428 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:40:17.149389    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-53a4f4/rate-test-model-53a4f4-kserve-7d76959979-pszlt"
Jul 23 15:40:17.149833 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:40:17.149446    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-53a4f4/rate-test-model-53a4f4-kserve-7d76959979-pszlt"
Jul 23 15:40:17.162711 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:40:17.162679    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-53a4f4/rate-test-model-53a4f4-kserve-7d76959979-pszlt"
Jul 23 15:40:17.969063 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:40:17.969029    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-53a4f4/rate-test-model-53a4f4-kserve-7d76959979-pszlt"
Jul 23 15:40:35.526714 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:40:35.526623    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cgwsg"]
Jul 23 15:40:50.425067 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:40:50.425030    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cgwsg"]
Jul 23 15:40:56.555882 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:40:56.555848    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-d3e0e7/rate-test-model-d3e0e7-kserve-8594b9d46c-rzh8k"]
Jul 23 15:40:57.540002 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:40:57.539966    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-53a4f4/rate-test-model-53a4f4-kserve-7d76959979-pszlt"]
Jul 23 15:41:07.923470 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:41:07.923434    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cgwsg"]
Jul 23 15:41:11.577170 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:41:11.577104    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-d3e0e7/rate-test-model-d3e0e7-kserve-8594b9d46c-rzh8k" podUID="848d6291-4db9-44d5-a57a-1b12c9a3d848" containerName="main" containerID="cri-o://53e5680acceeff6cb6b844dfd04809f34bf5b06ca4e3f67356be785cc435978e" gracePeriod=45
Jul 23 15:41:11.835137 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:41:11.835068    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-d3e0e7/rate-test-model-d3e0e7-kserve-8594b9d46c-rzh8k"
Jul 23 15:41:11.909461 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:41:11.909427    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mzn4z\" (UniqueName: \"kubernetes.io/projected/848d6291-4db9-44d5-a57a-1b12c9a3d848-kube-api-access-mzn4z\") pod \"848d6291-4db9-44d5-a57a-1b12c9a3d848\" (UID: \"848d6291-4db9-44d5-a57a-1b12c9a3d848\") "
Jul 23 15:41:11.909660 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:41:11.909482    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/848d6291-4db9-44d5-a57a-1b12c9a3d848-home\") pod \"848d6291-4db9-44d5-a57a-1b12c9a3d848\" (UID: \"848d6291-4db9-44d5-a57a-1b12c9a3d848\") "
Jul 23 15:41:11.909660 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:41:11.909514    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/848d6291-4db9-44d5-a57a-1b12c9a3d848-tmp-dir\") pod \"848d6291-4db9-44d5-a57a-1b12c9a3d848\" (UID: \"848d6291-4db9-44d5-a57a-1b12c9a3d848\") "
Jul 23 15:41:11.909660 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:41:11.909544    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/848d6291-4db9-44d5-a57a-1b12c9a3d848-model-cache\") pod \"848d6291-4db9-44d5-a57a-1b12c9a3d848\" (UID: \"848d6291-4db9-44d5-a57a-1b12c9a3d848\") "
Jul 23 15:41:11.909660 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:41:11.909565    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/848d6291-4db9-44d5-a57a-1b12c9a3d848-tls-certs\") pod \"848d6291-4db9-44d5-a57a-1b12c9a3d848\" (UID: \"848d6291-4db9-44d5-a57a-1b12c9a3d848\") "
Jul 23 15:41:11.909660 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:41:11.909588    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/848d6291-4db9-44d5-a57a-1b12c9a3d848-dshm\") pod \"848d6291-4db9-44d5-a57a-1b12c9a3d848\" (UID: \"848d6291-4db9-44d5-a57a-1b12c9a3d848\") "
Jul 23 15:41:11.909913 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:41:11.909821    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/848d6291-4db9-44d5-a57a-1b12c9a3d848-model-cache" (OuterVolumeSpecName: "model-cache") pod "848d6291-4db9-44d5-a57a-1b12c9a3d848" (UID: "848d6291-4db9-44d5-a57a-1b12c9a3d848"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 15:41:11.909913 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:41:11.909837    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/848d6291-4db9-44d5-a57a-1b12c9a3d848-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "848d6291-4db9-44d5-a57a-1b12c9a3d848" (UID: "848d6291-4db9-44d5-a57a-1b12c9a3d848"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 15:41:11.909913 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:41:11.909850    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/848d6291-4db9-44d5-a57a-1b12c9a3d848-home" (OuterVolumeSpecName: "home") pod "848d6291-4db9-44d5-a57a-1b12c9a3d848" (UID: "848d6291-4db9-44d5-a57a-1b12c9a3d848"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 15:41:11.911811 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:41:11.911790    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/848d6291-4db9-44d5-a57a-1b12c9a3d848-dshm" (OuterVolumeSpecName: "dshm") pod "848d6291-4db9-44d5-a57a-1b12c9a3d848" (UID: "848d6291-4db9-44d5-a57a-1b12c9a3d848"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 15:41:11.911910 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:41:11.911817    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/848d6291-4db9-44d5-a57a-1b12c9a3d848-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "848d6291-4db9-44d5-a57a-1b12c9a3d848" (UID: "848d6291-4db9-44d5-a57a-1b12c9a3d848"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 15:41:11.911958 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:41:11.911904    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/848d6291-4db9-44d5-a57a-1b12c9a3d848-kube-api-access-mzn4z" (OuterVolumeSpecName: "kube-api-access-mzn4z") pod "848d6291-4db9-44d5-a57a-1b12c9a3d848" (UID: "848d6291-4db9-44d5-a57a-1b12c9a3d848"). InnerVolumeSpecName "kube-api-access-mzn4z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 15:41:12.010720 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:41:12.010677    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/848d6291-4db9-44d5-a57a-1b12c9a3d848-tmp-dir\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Jul 23 15:41:12.010720 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:41:12.010711    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/848d6291-4db9-44d5-a57a-1b12c9a3d848-model-cache\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Jul 23 15:41:12.010720 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:41:12.010720    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/848d6291-4db9-44d5-a57a-1b12c9a3d848-tls-certs\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Jul 23 15:41:12.010720 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:41:12.010727    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/848d6291-4db9-44d5-a57a-1b12c9a3d848-dshm\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Jul 23 15:41:12.010720 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:41:12.010735    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzn4z\" (UniqueName: \"kubernetes.io/projected/848d6291-4db9-44d5-a57a-1b12c9a3d848-kube-api-access-mzn4z\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Jul 23 15:41:12.011018 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:41:12.010745    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/848d6291-4db9-44d5-a57a-1b12c9a3d848-home\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Jul 23 15:41:12.224701 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:41:12.224669    2560 generic.go:356] "Generic (PLEG): container finished" podID="848d6291-4db9-44d5-a57a-1b12c9a3d848" containerID="53e5680acceeff6cb6b844dfd04809f34bf5b06ca4e3f67356be785cc435978e" exitCode=0
Jul 23 15:41:12.224916 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:41:12.224715    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-d3e0e7/rate-test-model-d3e0e7-kserve-8594b9d46c-rzh8k"
Jul 23 15:41:12.224916 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:41:12.224760    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-d3e0e7/rate-test-model-d3e0e7-kserve-8594b9d46c-rzh8k" event={"ID":"848d6291-4db9-44d5-a57a-1b12c9a3d848","Type":"ContainerDied","Data":"53e5680acceeff6cb6b844dfd04809f34bf5b06ca4e3f67356be785cc435978e"}
Jul 23 15:41:12.224916 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:41:12.224808    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-d3e0e7/rate-test-model-d3e0e7-kserve-8594b9d46c-rzh8k" event={"ID":"848d6291-4db9-44d5-a57a-1b12c9a3d848","Type":"ContainerDied","Data":"72d08cd4553b8e862829ea33e6f6d7e3e301bd7f5375f60b74433cf35e89f0bf"}
Jul 23 15:41:12.224916 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:41:12.224828    2560 scope.go:117] "RemoveContainer" containerID="53e5680acceeff6cb6b844dfd04809f34bf5b06ca4e3f67356be785cc435978e"
Jul 23 15:41:12.234704 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:41:12.234688    2560 scope.go:117] "RemoveContainer" containerID="53e5680acceeff6cb6b844dfd04809f34bf5b06ca4e3f67356be785cc435978e"
Jul 23 15:41:12.234992 ip-10-0-141-105 kubenswrapper[2560]: E0723 15:41:12.234968    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53e5680acceeff6cb6b844dfd04809f34bf5b06ca4e3f67356be785cc435978e\": container with ID starting with 53e5680acceeff6cb6b844dfd04809f34bf5b06ca4e3f67356be785cc435978e not found: ID does not exist" containerID="53e5680acceeff6cb6b844dfd04809f34bf5b06ca4e3f67356be785cc435978e"
Jul 23 15:41:12.235056 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:41:12.235007    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53e5680acceeff6cb6b844dfd04809f34bf5b06ca4e3f67356be785cc435978e"} err="failed to get container status \"53e5680acceeff6cb6b844dfd04809f34bf5b06ca4e3f67356be785cc435978e\": rpc error: code = NotFound desc = could not find container \"53e5680acceeff6cb6b844dfd04809f34bf5b06ca4e3f67356be785cc435978e\": container with ID starting with 53e5680acceeff6cb6b844dfd04809f34bf5b06ca4e3f67356be785cc435978e not found: ID does not exist"
Jul 23 15:41:12.245766 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:41:12.245740    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-d3e0e7/rate-test-model-d3e0e7-kserve-8594b9d46c-rzh8k"]
Jul 23 15:41:12.249247 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:41:12.249204    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-d3e0e7/rate-test-model-d3e0e7-kserve-8594b9d46c-rzh8k"]
Jul 23 15:41:12.561085 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:41:12.560989    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-53a4f4/rate-test-model-53a4f4-kserve-7d76959979-pszlt" podUID="c81c1c6f-8867-41e7-b559-0adecfb7049e" containerName="main" containerID="cri-o://8d27eb68cbe51392960a48989c52f828dc07b5caf7da1b2661e6c3e404fd68c1" gracePeriod=45
Jul 23 15:41:12.811673 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:41:12.811615    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-53a4f4/rate-test-model-53a4f4-kserve-7d76959979-pszlt"
Jul 23 15:41:12.918088 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:41:12.918056    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c81c1c6f-8867-41e7-b559-0adecfb7049e-tls-certs\") pod \"c81c1c6f-8867-41e7-b559-0adecfb7049e\" (UID: \"c81c1c6f-8867-41e7-b559-0adecfb7049e\") "
Jul 23 15:41:12.918307 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:41:12.918111    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c81c1c6f-8867-41e7-b559-0adecfb7049e-dshm\") pod \"c81c1c6f-8867-41e7-b559-0adecfb7049e\" (UID: \"c81c1c6f-8867-41e7-b559-0adecfb7049e\") "
Jul 23 15:41:12.918307 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:41:12.918130    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c81c1c6f-8867-41e7-b559-0adecfb7049e-tmp-dir\") pod \"c81c1c6f-8867-41e7-b559-0adecfb7049e\" (UID: \"c81c1c6f-8867-41e7-b559-0adecfb7049e\") "
Jul 23 15:41:12.918307 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:41:12.918170    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c81c1c6f-8867-41e7-b559-0adecfb7049e-model-cache\") pod \"c81c1c6f-8867-41e7-b559-0adecfb7049e\" (UID: \"c81c1c6f-8867-41e7-b559-0adecfb7049e\") "
Jul 23 15:41:12.918307 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:41:12.918198    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z4p26\" (UniqueName: \"kubernetes.io/projected/c81c1c6f-8867-41e7-b559-0adecfb7049e-kube-api-access-z4p26\") pod \"c81c1c6f-8867-41e7-b559-0adecfb7049e\" (UID: \"c81c1c6f-8867-41e7-b559-0adecfb7049e\") "
Jul 23 15:41:12.918307 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:41:12.918222    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c81c1c6f-8867-41e7-b559-0adecfb7049e-home\") pod \"c81c1c6f-8867-41e7-b559-0adecfb7049e\" (UID: \"c81c1c6f-8867-41e7-b559-0adecfb7049e\") "
Jul 23 15:41:12.918599 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:41:12.918430    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c81c1c6f-8867-41e7-b559-0adecfb7049e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c81c1c6f-8867-41e7-b559-0adecfb7049e" (UID: "c81c1c6f-8867-41e7-b559-0adecfb7049e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 15:41:12.918599 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:41:12.918462    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c81c1c6f-8867-41e7-b559-0adecfb7049e-model-cache" (OuterVolumeSpecName: "model-cache") pod "c81c1c6f-8867-41e7-b559-0adecfb7049e" (UID: "c81c1c6f-8867-41e7-b559-0adecfb7049e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 15:41:12.918599 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:41:12.918551    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c81c1c6f-8867-41e7-b559-0adecfb7049e-tmp-dir\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Jul 23 15:41:12.918599 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:41:12.918578    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c81c1c6f-8867-41e7-b559-0adecfb7049e-model-cache\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Jul 23 15:41:12.918742 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:41:12.918597    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c81c1c6f-8867-41e7-b559-0adecfb7049e-home" (OuterVolumeSpecName: "home") pod "c81c1c6f-8867-41e7-b559-0adecfb7049e" (UID: "c81c1c6f-8867-41e7-b559-0adecfb7049e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 15:41:12.920478 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:41:12.920457    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c81c1c6f-8867-41e7-b559-0adecfb7049e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c81c1c6f-8867-41e7-b559-0adecfb7049e" (UID: "c81c1c6f-8867-41e7-b559-0adecfb7049e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 15:41:12.920877 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:41:12.920860    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c81c1c6f-8867-41e7-b559-0adecfb7049e-dshm" (OuterVolumeSpecName: "dshm") pod "c81c1c6f-8867-41e7-b559-0adecfb7049e" (UID: "c81c1c6f-8867-41e7-b559-0adecfb7049e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 15:41:12.920930 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:41:12.920901    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c81c1c6f-8867-41e7-b559-0adecfb7049e-kube-api-access-z4p26" (OuterVolumeSpecName: "kube-api-access-z4p26") pod "c81c1c6f-8867-41e7-b559-0adecfb7049e" (UID: "c81c1c6f-8867-41e7-b559-0adecfb7049e"). InnerVolumeSpecName "kube-api-access-z4p26". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 15:41:13.019681 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:41:13.019646    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4p26\" (UniqueName: \"kubernetes.io/projected/c81c1c6f-8867-41e7-b559-0adecfb7049e-kube-api-access-z4p26\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Jul 23 15:41:13.019681 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:41:13.019676    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c81c1c6f-8867-41e7-b559-0adecfb7049e-home\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Jul 23 15:41:13.019681 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:41:13.019686    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c81c1c6f-8867-41e7-b559-0adecfb7049e-tls-certs\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Jul 23 15:41:13.019919 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:41:13.019699    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c81c1c6f-8867-41e7-b559-0adecfb7049e-dshm\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Jul 23 15:41:13.062276 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:41:13.062176    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="848d6291-4db9-44d5-a57a-1b12c9a3d848" path="/var/lib/kubelet/pods/848d6291-4db9-44d5-a57a-1b12c9a3d848/volumes"
Jul 23 15:41:13.231433 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:41:13.231399    2560 generic.go:356] "Generic (PLEG): container finished" podID="c81c1c6f-8867-41e7-b559-0adecfb7049e" containerID="8d27eb68cbe51392960a48989c52f828dc07b5caf7da1b2661e6c3e404fd68c1" exitCode=0
Jul 23 15:41:13.231433 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:41:13.231437    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-53a4f4/rate-test-model-53a4f4-kserve-7d76959979-pszlt" event={"ID":"c81c1c6f-8867-41e7-b559-0adecfb7049e","Type":"ContainerDied","Data":"8d27eb68cbe51392960a48989c52f828dc07b5caf7da1b2661e6c3e404fd68c1"}
Jul 23 15:41:13.231661 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:41:13.231444    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-53a4f4/rate-test-model-53a4f4-kserve-7d76959979-pszlt"
Jul 23 15:41:13.231661 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:41:13.231463    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-53a4f4/rate-test-model-53a4f4-kserve-7d76959979-pszlt" event={"ID":"c81c1c6f-8867-41e7-b559-0adecfb7049e","Type":"ContainerDied","Data":"cc801f56cc1d67619cac85bc2d48febf427eb9962259de370689f4a0aec33f95"}
Jul 23 15:41:13.231661 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:41:13.231483    2560 scope.go:117] "RemoveContainer" containerID="8d27eb68cbe51392960a48989c52f828dc07b5caf7da1b2661e6c3e404fd68c1"
Jul 23 15:41:13.240555 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:41:13.240536    2560 scope.go:117] "RemoveContainer" containerID="8d27eb68cbe51392960a48989c52f828dc07b5caf7da1b2661e6c3e404fd68c1"
Jul 23 15:41:13.240812 ip-10-0-141-105 kubenswrapper[2560]: E0723 15:41:13.240794    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d27eb68cbe51392960a48989c52f828dc07b5caf7da1b2661e6c3e404fd68c1\": container with ID starting with 8d27eb68cbe51392960a48989c52f828dc07b5caf7da1b2661e6c3e404fd68c1 not found: ID does not exist" containerID="8d27eb68cbe51392960a48989c52f828dc07b5caf7da1b2661e6c3e404fd68c1"
Jul 23 15:41:13.240876 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:41:13.240825    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d27eb68cbe51392960a48989c52f828dc07b5caf7da1b2661e6c3e404fd68c1"} err="failed to get container status \"8d27eb68cbe51392960a48989c52f828dc07b5caf7da1b2661e6c3e404fd68c1\": rpc error: code = NotFound desc = could not find container \"8d27eb68cbe51392960a48989c52f828dc07b5caf7da1b2661e6c3e404fd68c1\": container with ID starting with 8d27eb68cbe51392960a48989c52f828dc07b5caf7da1b2661e6c3e404fd68c1 not found: ID does not exist"
Jul 23 15:41:13.247532 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:41:13.247508    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-53a4f4/rate-test-model-53a4f4-kserve-7d76959979-pszlt"]
Jul 23 15:41:13.251012 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:41:13.250991    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-53a4f4/rate-test-model-53a4f4-kserve-7d76959979-pszlt"]
Jul 23 15:41:15.062718 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:41:15.062682    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c81c1c6f-8867-41e7-b559-0adecfb7049e" path="/var/lib/kubelet/pods/c81c1c6f-8867-41e7-b559-0adecfb7049e/volumes"
Jul 23 15:41:27.801699 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:41:27.801661    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-d3e0e7-66b6648864-dqvjm"]
Jul 23 15:41:27.802165 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:41:27.801874    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d3e0e7-66b6648864-dqvjm" podUID="9289cc95-30b3-4bc8-b933-4232fc87cbf9" containerName="maas-api" containerID="cri-o://f05da802f817451f6a802c454f5699ee13099f6d61a03029aa45460a95a81149" gracePeriod=30
Jul 23 15:41:28.248280 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:41:28.248221    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d3e0e7-66b6648864-dqvjm"
Jul 23 15:41:28.303442 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:41:28.303407    2560 generic.go:356] "Generic (PLEG): container finished" podID="9289cc95-30b3-4bc8-b933-4232fc87cbf9" containerID="f05da802f817451f6a802c454f5699ee13099f6d61a03029aa45460a95a81149" exitCode=0
Jul 23 15:41:28.303631 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:41:28.303457    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d3e0e7-66b6648864-dqvjm"
Jul 23 15:41:28.303631 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:41:28.303486    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d3e0e7-66b6648864-dqvjm" event={"ID":"9289cc95-30b3-4bc8-b933-4232fc87cbf9","Type":"ContainerDied","Data":"f05da802f817451f6a802c454f5699ee13099f6d61a03029aa45460a95a81149"}
Jul 23 15:41:28.303631 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:41:28.303538    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d3e0e7-66b6648864-dqvjm" event={"ID":"9289cc95-30b3-4bc8-b933-4232fc87cbf9","Type":"ContainerDied","Data":"16e959ac10affc7caabb143e8b39437838f0ccf0d81809d5388704a7625448f3"}
Jul 23 15:41:28.303631 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:41:28.303561    2560 scope.go:117] "RemoveContainer" containerID="f05da802f817451f6a802c454f5699ee13099f6d61a03029aa45460a95a81149"
Jul 23 15:41:28.313982 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:41:28.313959    2560 scope.go:117] "RemoveContainer" containerID="f05da802f817451f6a802c454f5699ee13099f6d61a03029aa45460a95a81149"
Jul 23 15:41:28.314304 ip-10-0-141-105 kubenswrapper[2560]: E0723 15:41:28.314282    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f05da802f817451f6a802c454f5699ee13099f6d61a03029aa45460a95a81149\": container with ID starting with f05da802f817451f6a802c454f5699ee13099f6d61a03029aa45460a95a81149 not found: ID does not exist" containerID="f05da802f817451f6a802c454f5699ee13099f6d61a03029aa45460a95a81149"
Jul 23 15:41:28.314378 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:41:28.314319    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f05da802f817451f6a802c454f5699ee13099f6d61a03029aa45460a95a81149"} err="failed to get container status \"f05da802f817451f6a802c454f5699ee13099f6d61a03029aa45460a95a81149\": rpc error: code = NotFound desc = could not find container \"f05da802f817451f6a802c454f5699ee13099f6d61a03029aa45460a95a81149\": container with ID starting with f05da802f817451f6a802c454f5699ee13099f6d61a03029aa45460a95a81149 not found: ID does not exist"
Jul 23 15:41:28.358479 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:41:28.358390    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9289cc95-30b3-4bc8-b933-4232fc87cbf9-openshift-service-ca\") pod \"9289cc95-30b3-4bc8-b933-4232fc87cbf9\" (UID: \"9289cc95-30b3-4bc8-b933-4232fc87cbf9\") "
Jul 23 15:41:28.358479 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:41:28.358448    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5g497\" (UniqueName: \"kubernetes.io/projected/9289cc95-30b3-4bc8-b933-4232fc87cbf9-kube-api-access-5g497\") pod \"9289cc95-30b3-4bc8-b933-4232fc87cbf9\" (UID: \"9289cc95-30b3-4bc8-b933-4232fc87cbf9\") "
Jul 23 15:41:28.358696 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:41:28.358484    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9289cc95-30b3-4bc8-b933-4232fc87cbf9-maas-api-tls\") pod \"9289cc95-30b3-4bc8-b933-4232fc87cbf9\" (UID: \"9289cc95-30b3-4bc8-b933-4232fc87cbf9\") "
Jul 23 15:41:28.358814 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:41:28.358789    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9289cc95-30b3-4bc8-b933-4232fc87cbf9-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9289cc95-30b3-4bc8-b933-4232fc87cbf9" (UID: "9289cc95-30b3-4bc8-b933-4232fc87cbf9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 15:41:28.360685 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:41:28.360658    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9289cc95-30b3-4bc8-b933-4232fc87cbf9-kube-api-access-5g497" (OuterVolumeSpecName: "kube-api-access-5g497") pod "9289cc95-30b3-4bc8-b933-4232fc87cbf9" (UID: "9289cc95-30b3-4bc8-b933-4232fc87cbf9"). InnerVolumeSpecName "kube-api-access-5g497". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 15:41:28.360685 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:41:28.360674    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9289cc95-30b3-4bc8-b933-4232fc87cbf9-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9289cc95-30b3-4bc8-b933-4232fc87cbf9" (UID: "9289cc95-30b3-4bc8-b933-4232fc87cbf9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 15:41:28.460068 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:41:28.460029    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9289cc95-30b3-4bc8-b933-4232fc87cbf9-maas-api-tls\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Jul 23 15:41:28.460068 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:41:28.460066    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9289cc95-30b3-4bc8-b933-4232fc87cbf9-openshift-service-ca\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Jul 23 15:41:28.460068 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:41:28.460078    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5g497\" (UniqueName: \"kubernetes.io/projected/9289cc95-30b3-4bc8-b933-4232fc87cbf9-kube-api-access-5g497\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Jul 23 15:41:28.624284 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:41:28.624250    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-d3e0e7-66b6648864-dqvjm"]
Jul 23 15:41:28.629561 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:41:28.629536    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-d3e0e7-66b6648864-dqvjm"]
Jul 23 15:41:29.062878 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:41:29.062842    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9289cc95-30b3-4bc8-b933-4232fc87cbf9" path="/var/lib/kubelet/pods/9289cc95-30b3-4bc8-b933-4232fc87cbf9/volumes"
Jul 23 15:41:30.925298 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:41:30.925259    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cgwsg"]
Jul 23 15:41:49.612547 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:41:49.612511    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-53a4f4-786d7c49bd-4wjjh"]
Jul 23 15:41:49.612961 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:41:49.612734    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-53a4f4-786d7c49bd-4wjjh" podUID="f084c91f-3708-4715-b7aa-b176518042bc" containerName="maas-api" containerID="cri-o://804ff05cba99377e263871b47cbd7de11e10046d1d544181417c08d14ee3a53b" gracePeriod=30
Jul 23 15:41:49.868444 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:41:49.868381    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-53a4f4-786d7c49bd-4wjjh"
Jul 23 15:41:50.052455 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:41:50.052418    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f084c91f-3708-4715-b7aa-b176518042bc-openshift-service-ca\") pod \"f084c91f-3708-4715-b7aa-b176518042bc\" (UID: \"f084c91f-3708-4715-b7aa-b176518042bc\") "
Jul 23 15:41:50.052455 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:41:50.052465    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tt69q\" (UniqueName: \"kubernetes.io/projected/f084c91f-3708-4715-b7aa-b176518042bc-kube-api-access-tt69q\") pod \"f084c91f-3708-4715-b7aa-b176518042bc\" (UID: \"f084c91f-3708-4715-b7aa-b176518042bc\") "
Jul 23 15:41:50.052712 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:41:50.052565    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f084c91f-3708-4715-b7aa-b176518042bc-maas-api-tls\") pod \"f084c91f-3708-4715-b7aa-b176518042bc\" (UID: \"f084c91f-3708-4715-b7aa-b176518042bc\") "
Jul 23 15:41:50.052863 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:41:50.052833    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f084c91f-3708-4715-b7aa-b176518042bc-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f084c91f-3708-4715-b7aa-b176518042bc" (UID: "f084c91f-3708-4715-b7aa-b176518042bc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 15:41:50.054746 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:41:50.054723    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f084c91f-3708-4715-b7aa-b176518042bc-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f084c91f-3708-4715-b7aa-b176518042bc" (UID: "f084c91f-3708-4715-b7aa-b176518042bc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 15:41:50.054857 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:41:50.054835    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f084c91f-3708-4715-b7aa-b176518042bc-kube-api-access-tt69q" (OuterVolumeSpecName: "kube-api-access-tt69q") pod "f084c91f-3708-4715-b7aa-b176518042bc" (UID: "f084c91f-3708-4715-b7aa-b176518042bc"). InnerVolumeSpecName "kube-api-access-tt69q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 15:41:50.154125 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:41:50.154032    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f084c91f-3708-4715-b7aa-b176518042bc-maas-api-tls\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Jul 23 15:41:50.154125 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:41:50.154064    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f084c91f-3708-4715-b7aa-b176518042bc-openshift-service-ca\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Jul 23 15:41:50.154125 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:41:50.154079    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tt69q\" (UniqueName: \"kubernetes.io/projected/f084c91f-3708-4715-b7aa-b176518042bc-kube-api-access-tt69q\") on node \"ip-10-0-141-105.ec2.internal\" DevicePath \"\""
Jul 23 15:41:50.406771 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:41:50.406678    2560 generic.go:356] "Generic (PLEG): container finished" podID="f084c91f-3708-4715-b7aa-b176518042bc" containerID="804ff05cba99377e263871b47cbd7de11e10046d1d544181417c08d14ee3a53b" exitCode=0
Jul 23 15:41:50.406771 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:41:50.406726    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-53a4f4-786d7c49bd-4wjjh"
Jul 23 15:41:50.406950 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:41:50.406769    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-53a4f4-786d7c49bd-4wjjh" event={"ID":"f084c91f-3708-4715-b7aa-b176518042bc","Type":"ContainerDied","Data":"804ff05cba99377e263871b47cbd7de11e10046d1d544181417c08d14ee3a53b"}
Jul 23 15:41:50.406950 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:41:50.406824    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-53a4f4-786d7c49bd-4wjjh" event={"ID":"f084c91f-3708-4715-b7aa-b176518042bc","Type":"ContainerDied","Data":"7bb87c8c186ca2238c84332d58d574b00f1fab57bef7f6f8ebe1265a0aac22e3"}
Jul 23 15:41:50.406950 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:41:50.406847    2560 scope.go:117] "RemoveContainer" containerID="804ff05cba99377e263871b47cbd7de11e10046d1d544181417c08d14ee3a53b"
Jul 23 15:41:50.416163 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:41:50.416147    2560 scope.go:117] "RemoveContainer" containerID="804ff05cba99377e263871b47cbd7de11e10046d1d544181417c08d14ee3a53b"
Jul 23 15:41:50.416425 ip-10-0-141-105 kubenswrapper[2560]: E0723 15:41:50.416401    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"804ff05cba99377e263871b47cbd7de11e10046d1d544181417c08d14ee3a53b\": container with ID starting with 804ff05cba99377e263871b47cbd7de11e10046d1d544181417c08d14ee3a53b not found: ID does not exist" containerID="804ff05cba99377e263871b47cbd7de11e10046d1d544181417c08d14ee3a53b"
Jul 23 15:41:50.416495 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:41:50.416436    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"804ff05cba99377e263871b47cbd7de11e10046d1d544181417c08d14ee3a53b"} err="failed to get container status \"804ff05cba99377e263871b47cbd7de11e10046d1d544181417c08d14ee3a53b\": rpc error: code = NotFound desc = could not find container \"804ff05cba99377e263871b47cbd7de11e10046d1d544181417c08d14ee3a53b\": container with ID starting with 804ff05cba99377e263871b47cbd7de11e10046d1d544181417c08d14ee3a53b not found: ID does not exist"
Jul 23 15:41:50.427675 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:41:50.427649    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-53a4f4-786d7c49bd-4wjjh"]
Jul 23 15:41:50.431734 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:41:50.431713    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-53a4f4-786d7c49bd-4wjjh"]
Jul 23 15:41:51.062031 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:41:51.061997    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f084c91f-3708-4715-b7aa-b176518042bc" path="/var/lib/kubelet/pods/f084c91f-3708-4715-b7aa-b176518042bc/volumes"
Jul 23 15:41:52.523202 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:41:52.523168    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cgwsg"]
Jul 23 15:42:01.450995 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:42:01.450873    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6777b9bd49-vb587_36f2ed03-9cc1-421b-821a-140db7c0a1c6/authorino/0.log"
Jul 23 15:42:08.375473 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:42:08.375439    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-56b97c94c-nb24s_0090c8f7-34af-4caa-a4ad-6cb6d6152380/copy-manifests/0.log"
Jul 23 15:42:08.382922 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:42:08.382896    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-56b97c94c-nb24s_0090c8f7-34af-4caa-a4ad-6cb6d6152380/manager/0.log"
Jul 23 15:42:10.912150 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:42:10.912115    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6777b9bd49-vb587_36f2ed03-9cc1-421b-821a-140db7c0a1c6/authorino/0.log"
Jul 23 15:42:11.212863 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:42:11.212783    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-w4bvt_fd4a0407-3631-43d7-8632-d5382863958f/manager/0.log"
Jul 23 15:42:11.334465 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:42:11.334433    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-g6n6s_c125c2da-4370-4e38-9a16-c147b743a797/kuadrant-console-plugin/0.log"
Jul 23 15:42:11.864854 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:42:11.864825    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-cgwsg_73e0120d-710b-45eb-9a73-83729e5a4c4a/limitador/0.log"
Jul 23 15:42:13.916009 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:42:13.915979    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-8d89f64f-gx5q6_e08125f5-7cbf-4bf3-a666-56b514d5b964/kube-auth-proxy/0.log"
Jul 23 15:42:15.379028 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:42:15.378981    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-9htj9_36060547-3588-4f78-b737-714d88f73dc3/main/0.log"
Jul 23 15:42:15.780072 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:42:15.780039    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-zvpnp_22c9c0ea-6dc7-4915-bc33-ab6ace48274e/main/0.log"
Jul 23 15:42:27.042752 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:42:27.042723    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-hvsnb_527775ad-7c13-4732-b824-4302ad79dcf4/global-pull-secret-syncer/0.log"
Jul 23 15:42:27.183739 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:42:27.183707    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-ns8qd_b3df911b-3c63-441b-97fa-8d431bf3fb95/konnectivity-agent/0.log"
Jul 23 15:42:27.205549 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:42:27.205523    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-105.ec2.internal_3d37fd728dd468b7ee1c820042e75b2d/haproxy/0.log"
Jul 23 15:42:32.562498 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:42:32.562458    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6777b9bd49-vb587_36f2ed03-9cc1-421b-821a-140db7c0a1c6/authorino/0.log"
Jul 23 15:42:32.648739 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:42:32.648709    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-w4bvt_fd4a0407-3631-43d7-8632-d5382863958f/manager/0.log"
Jul 23 15:42:32.686138 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:42:32.686108    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-g6n6s_c125c2da-4370-4e38-9a16-c147b743a797/kuadrant-console-plugin/0.log"
Jul 23 15:42:32.980005 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:42:32.979979    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-cgwsg_73e0120d-710b-45eb-9a73-83729e5a4c4a/limitador/0.log"
Jul 23 15:42:34.705043 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:42:34.704994    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_766a6fa8-2283-4a2f-9f94-9ccfeaa59434/alertmanager/0.log"
Jul 23 15:42:34.729520 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:42:34.729489    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_766a6fa8-2283-4a2f-9f94-9ccfeaa59434/config-reloader/0.log"
Jul 23 15:42:34.750518 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:42:34.750484    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_766a6fa8-2283-4a2f-9f94-9ccfeaa59434/kube-rbac-proxy-web/0.log"
Jul 23 15:42:34.772930 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:42:34.772908    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_766a6fa8-2283-4a2f-9f94-9ccfeaa59434/kube-rbac-proxy/0.log"
Jul 23 15:42:34.792967 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:42:34.792931    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_766a6fa8-2283-4a2f-9f94-9ccfeaa59434/kube-rbac-proxy-metric/0.log"
Jul 23 15:42:34.811807 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:42:34.811783    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_766a6fa8-2283-4a2f-9f94-9ccfeaa59434/prom-label-proxy/0.log"
Jul 23 15:42:34.831073 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:42:34.831041    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_766a6fa8-2283-4a2f-9f94-9ccfeaa59434/init-config-reloader/0.log"
Jul 23 15:42:35.074836 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:42:35.074812    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-7b7b7bc8bf-29hwq_e1ee8253-0c66-4e49-827b-42d03d36c796/monitoring-plugin/0.log"
Jul 23 15:42:35.251443 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:42:35.251410    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-wt7xl_a2e2585d-0204-4faa-807e-b1e28263c9bf/node-exporter/0.log"
Jul 23 15:42:35.285974 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:42:35.285864    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-wt7xl_a2e2585d-0204-4faa-807e-b1e28263c9bf/kube-rbac-proxy/0.log"
Jul 23 15:42:35.316883 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:42:35.316854    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-wt7xl_a2e2585d-0204-4faa-807e-b1e28263c9bf/init-textfile/0.log"
Jul 23 15:42:35.520579 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:42:35.520549    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_da9c516e-f152-474a-a04a-bfbdabaeb022/prometheus/0.log"
Jul 23 15:42:35.551409 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:42:35.551321    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_da9c516e-f152-474a-a04a-bfbdabaeb022/config-reloader/0.log"
Jul 23 15:42:35.589194 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:42:35.589172    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_da9c516e-f152-474a-a04a-bfbdabaeb022/thanos-sidecar/0.log"
Jul 23 15:42:35.627720 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:42:35.627695    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_da9c516e-f152-474a-a04a-bfbdabaeb022/kube-rbac-proxy-web/0.log"
Jul 23 15:42:35.656177 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:42:35.656137    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_da9c516e-f152-474a-a04a-bfbdabaeb022/kube-rbac-proxy/0.log"
Jul 23 15:42:35.683375 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:42:35.683349    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_da9c516e-f152-474a-a04a-bfbdabaeb022/kube-rbac-proxy-thanos/0.log"
Jul 23 15:42:35.723842 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:42:35.723815    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_da9c516e-f152-474a-a04a-bfbdabaeb022/init-config-reloader/0.log"
Jul 23 15:42:37.175438 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:42:37.175401    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-8lzsc/perf-node-gather-daemonset-h5vsn"]
Jul 23 15:42:37.175826 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:42:37.175743    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f084c91f-3708-4715-b7aa-b176518042bc" containerName="maas-api"
Jul 23 15:42:37.175826 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:42:37.175754    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="f084c91f-3708-4715-b7aa-b176518042bc" containerName="maas-api"
Jul 23 15:42:37.175826 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:42:37.175768    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="848d6291-4db9-44d5-a57a-1b12c9a3d848" containerName="main"
Jul 23 15:42:37.175826 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:42:37.175773    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="848d6291-4db9-44d5-a57a-1b12c9a3d848" containerName="main"
Jul 23 15:42:37.175826 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:42:37.175781    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c81c1c6f-8867-41e7-b559-0adecfb7049e" containerName="main"
Jul 23 15:42:37.175826 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:42:37.175787    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="c81c1c6f-8867-41e7-b559-0adecfb7049e" containerName="main"
Jul 23 15:42:37.175826 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:42:37.175796    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9289cc95-30b3-4bc8-b933-4232fc87cbf9" containerName="maas-api"
Jul 23 15:42:37.175826 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:42:37.175801    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="9289cc95-30b3-4bc8-b933-4232fc87cbf9" containerName="maas-api"
Jul 23 15:42:37.176073 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:42:37.175865    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="848d6291-4db9-44d5-a57a-1b12c9a3d848" containerName="main"
Jul 23 15:42:37.176073 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:42:37.175872    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="f084c91f-3708-4715-b7aa-b176518042bc" containerName="maas-api"
Jul 23 15:42:37.176073 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:42:37.175878    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="9289cc95-30b3-4bc8-b933-4232fc87cbf9" containerName="maas-api"
Jul 23 15:42:37.176073 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:42:37.175886    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="c81c1c6f-8867-41e7-b559-0adecfb7049e" containerName="main"
Jul 23 15:42:37.179224 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:42:37.179207    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8lzsc/perf-node-gather-daemonset-h5vsn"
Jul 23 15:42:37.181833 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:42:37.181808    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-8lzsc\"/\"openshift-service-ca.crt\""
Jul 23 15:42:37.181956 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:42:37.181808    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-8lzsc\"/\"default-dockercfg-nn8hv\""
Jul 23 15:42:37.183010 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:42:37.182987    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-8lzsc\"/\"kube-root-ca.crt\""
Jul 23 15:42:37.185666 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:42:37.185640    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8lzsc/perf-node-gather-daemonset-h5vsn"]
Jul 23 15:42:37.367191 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:42:37.367153    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0c832659-0285-4ead-be28-3728bd7c898f-sys\") pod \"perf-node-gather-daemonset-h5vsn\" (UID: \"0c832659-0285-4ead-be28-3728bd7c898f\") " pod="openshift-must-gather-8lzsc/perf-node-gather-daemonset-h5vsn"
Jul 23 15:42:37.367388 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:42:37.367198    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkvvz\" (UniqueName: \"kubernetes.io/projected/0c832659-0285-4ead-be28-3728bd7c898f-kube-api-access-gkvvz\") pod \"perf-node-gather-daemonset-h5vsn\" (UID: \"0c832659-0285-4ead-be28-3728bd7c898f\") " pod="openshift-must-gather-8lzsc/perf-node-gather-daemonset-h5vsn"
Jul 23 15:42:37.367388 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:42:37.367326    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0c832659-0285-4ead-be28-3728bd7c898f-proc\") pod \"perf-node-gather-daemonset-h5vsn\" (UID: \"0c832659-0285-4ead-be28-3728bd7c898f\") " pod="openshift-must-gather-8lzsc/perf-node-gather-daemonset-h5vsn"
Jul 23 15:42:37.367465 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:42:37.367388    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0c832659-0285-4ead-be28-3728bd7c898f-lib-modules\") pod \"perf-node-gather-daemonset-h5vsn\" (UID: \"0c832659-0285-4ead-be28-3728bd7c898f\") " pod="openshift-must-gather-8lzsc/perf-node-gather-daemonset-h5vsn"
Jul 23 15:42:37.367465 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:42:37.367420    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0c832659-0285-4ead-be28-3728bd7c898f-podres\") pod \"perf-node-gather-daemonset-h5vsn\" (UID: \"0c832659-0285-4ead-be28-3728bd7c898f\") " pod="openshift-must-gather-8lzsc/perf-node-gather-daemonset-h5vsn"
Jul 23 15:42:37.468544 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:42:37.468507    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0c832659-0285-4ead-be28-3728bd7c898f-sys\") pod \"perf-node-gather-daemonset-h5vsn\" (UID: \"0c832659-0285-4ead-be28-3728bd7c898f\") " pod="openshift-must-gather-8lzsc/perf-node-gather-daemonset-h5vsn"
Jul 23 15:42:37.468544 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:42:37.468550    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkvvz\" (UniqueName: \"kubernetes.io/projected/0c832659-0285-4ead-be28-3728bd7c898f-kube-api-access-gkvvz\") pod \"perf-node-gather-daemonset-h5vsn\" (UID: \"0c832659-0285-4ead-be28-3728bd7c898f\") " pod="openshift-must-gather-8lzsc/perf-node-gather-daemonset-h5vsn"
Jul 23 15:42:37.468765 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:42:37.468651    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0c832659-0285-4ead-be28-3728bd7c898f-sys\") pod \"perf-node-gather-daemonset-h5vsn\" (UID: \"0c832659-0285-4ead-be28-3728bd7c898f\") " pod="openshift-must-gather-8lzsc/perf-node-gather-daemonset-h5vsn"
Jul 23 15:42:37.468765 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:42:37.468690    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0c832659-0285-4ead-be28-3728bd7c898f-proc\") pod \"perf-node-gather-daemonset-h5vsn\" (UID: \"0c832659-0285-4ead-be28-3728bd7c898f\") " pod="openshift-must-gather-8lzsc/perf-node-gather-daemonset-h5vsn"
Jul 23 15:42:37.468765 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:42:37.468755    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0c832659-0285-4ead-be28-3728bd7c898f-lib-modules\") pod \"perf-node-gather-daemonset-h5vsn\" (UID: \"0c832659-0285-4ead-be28-3728bd7c898f\") " pod="openshift-must-gather-8lzsc/perf-node-gather-daemonset-h5vsn"
Jul 23 15:42:37.468891 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:42:37.468781    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0c832659-0285-4ead-be28-3728bd7c898f-podres\") pod \"perf-node-gather-daemonset-h5vsn\" (UID: \"0c832659-0285-4ead-be28-3728bd7c898f\") " pod="openshift-must-gather-8lzsc/perf-node-gather-daemonset-h5vsn"
Jul 23 15:42:37.468891 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:42:37.468785    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0c832659-0285-4ead-be28-3728bd7c898f-proc\") pod \"perf-node-gather-daemonset-h5vsn\" (UID: \"0c832659-0285-4ead-be28-3728bd7c898f\") " pod="openshift-must-gather-8lzsc/perf-node-gather-daemonset-h5vsn"
Jul 23 15:42:37.468970 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:42:37.468919    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0c832659-0285-4ead-be28-3728bd7c898f-lib-modules\") pod \"perf-node-gather-daemonset-h5vsn\" (UID: \"0c832659-0285-4ead-be28-3728bd7c898f\") " pod="openshift-must-gather-8lzsc/perf-node-gather-daemonset-h5vsn"
Jul 23 15:42:37.468970 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:42:37.468932    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0c832659-0285-4ead-be28-3728bd7c898f-podres\") pod \"perf-node-gather-daemonset-h5vsn\" (UID: \"0c832659-0285-4ead-be28-3728bd7c898f\") " pod="openshift-must-gather-8lzsc/perf-node-gather-daemonset-h5vsn"
Jul 23 15:42:37.476676 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:42:37.476655    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkvvz\" (UniqueName: \"kubernetes.io/projected/0c832659-0285-4ead-be28-3728bd7c898f-kube-api-access-gkvvz\") pod \"perf-node-gather-daemonset-h5vsn\" (UID: \"0c832659-0285-4ead-be28-3728bd7c898f\") " pod="openshift-must-gather-8lzsc/perf-node-gather-daemonset-h5vsn"
Jul 23 15:42:37.490676 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:42:37.490653    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8lzsc/perf-node-gather-daemonset-h5vsn"
Jul 23 15:42:37.623829 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:42:37.623805    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8lzsc/perf-node-gather-daemonset-h5vsn"]
Jul 23 15:42:37.626693 ip-10-0-141-105 kubenswrapper[2560]: W0723 15:42:37.626669    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod0c832659_0285_4ead_be28_3728bd7c898f.slice/crio-688b9717edbbd88d0d2cb4be7af303eda19882e8a16b226e7837f25209a44fae WatchSource:0}: Error finding container 688b9717edbbd88d0d2cb4be7af303eda19882e8a16b226e7837f25209a44fae: Status 404 returned error can't find the container with id 688b9717edbbd88d0d2cb4be7af303eda19882e8a16b226e7837f25209a44fae
Jul 23 15:42:38.427102 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:42:38.427069    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-5bc67dcd46-rzmqx_ea3c4e5e-ccb4-417f-a9d9-5edc359ea988/download-server/0.log"
Jul 23 15:42:38.628854 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:42:38.628821    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8lzsc/perf-node-gather-daemonset-h5vsn" event={"ID":"0c832659-0285-4ead-be28-3728bd7c898f","Type":"ContainerStarted","Data":"cfe29d0058b32bcac226d72f3ac5f09dc3e85b0a9e6621b6451d7f52d70b9690"}
Jul 23 15:42:38.629042 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:42:38.628873    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8lzsc/perf-node-gather-daemonset-h5vsn" event={"ID":"0c832659-0285-4ead-be28-3728bd7c898f","Type":"ContainerStarted","Data":"688b9717edbbd88d0d2cb4be7af303eda19882e8a16b226e7837f25209a44fae"}
Jul 23 15:42:38.629042 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:42:38.628991    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-8lzsc/perf-node-gather-daemonset-h5vsn"
Jul 23 15:42:38.646002 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:42:38.645958    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-8lzsc/perf-node-gather-daemonset-h5vsn" podStartSLOduration=1.645942803 podStartE2EDuration="1.645942803s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 15:42:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 15:42:38.644454383 +0000 UTC m=+3000.152328131" watchObservedRunningTime="2026-07-23 15:42:38.645942803 +0000 UTC m=+3000.153816552"
Jul 23 15:42:40.094638 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:42:40.094607    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-scspg_a2a2b51d-3d22-42dd-96b6-093bdf7c3f7a/dns/0.log"
Jul 23 15:42:40.114444 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:42:40.114418    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-scspg_a2a2b51d-3d22-42dd-96b6-093bdf7c3f7a/kube-rbac-proxy/0.log"
Jul 23 15:42:40.180120 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:42:40.180091    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-vfsh6_d9b3550d-e82f-4934-b093-486563f240fb/dns-node-resolver/0.log"
Jul 23 15:42:40.375057 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:42:40.374894    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z4fvn_e0c66f26-06fe-4273-ba29-fa20ce89b873/ovn-acl-logging/0.log"
Jul 23 15:42:40.394249 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:42:40.385166    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z4fvn_e0c66f26-06fe-4273-ba29-fa20ce89b873/ovn-acl-logging/0.log"
Jul 23 15:42:40.772266 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:42:40.772217    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-lz8qd_9a2b87d6-d3dd-41b8-96f8-2e7a0be09325/node-ca/0.log"
Jul 23 15:42:41.811848 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:42:41.811825    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-8d89f64f-gx5q6_e08125f5-7cbf-4bf3-a666-56b514d5b964/kube-auth-proxy/0.log"
Jul 23 15:42:42.710241 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:42:42.710197    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-sw5j8_f7786ed8-2ad5-40ad-a866-ddacdead5ab1/serve-healthcheck-canary/0.log"
Jul 23 15:42:43.229445 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:42:43.229413    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-555d9d7885-zndrp_a253789c-c1c0-409f-b335-63fe5f2fb4af/insights-operator/0.log"
Jul 23 15:42:43.229865 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:42:43.229568    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-555d9d7885-zndrp_a253789c-c1c0-409f-b335-63fe5f2fb4af/insights-operator/1.log"
Jul 23 15:42:43.252019 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:42:43.251995    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5xjtm_823379c6-9443-4acd-9c9f-8c300490b6b2/kube-rbac-proxy/0.log"
Jul 23 15:42:43.271818 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:42:43.271793    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5xjtm_823379c6-9443-4acd-9c9f-8c300490b6b2/exporter/0.log"
Jul 23 15:42:43.292291 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:42:43.292268    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5xjtm_823379c6-9443-4acd-9c9f-8c300490b6b2/extractor/0.log"
Jul 23 15:42:44.644279 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:42:44.644224    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-8lzsc/perf-node-gather-daemonset-h5vsn"
Jul 23 15:42:46.047099 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:42:46.047070    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-56b97c94c-nb24s_0090c8f7-34af-4caa-a4ad-6cb6d6152380/manager/0.log"
Jul 23 15:42:46.067421 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:42:46.067399    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-56b97c94c-nb24s_0090c8f7-34af-4caa-a4ad-6cb6d6152380/copy-manifests/0.log"
Jul 23 15:42:47.536480 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:42:47.536407    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-5857c6d56c-kqj6p_e38c06a3-ef37-4429-8198-72de28028b71/manager/0.log"
Jul 23 15:42:47.960000 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:42:47.959918    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-8d89f64f-gx5q6_e08125f5-7cbf-4bf3-a666-56b514d5b964/kube-auth-proxy/0.log"
Jul 23 15:42:49.421031 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:42:49.420999    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-8d89f64f-gx5q6_e08125f5-7cbf-4bf3-a666-56b514d5b964/kube-auth-proxy/0.log"
Jul 23 15:42:52.843857 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:42:52.843825    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7658d9b749-k5zd5_9ae597eb-68e0-447c-bf0c-781a80684af1/kube-storage-version-migrator-operator/1.log"
Jul 23 15:42:52.844924 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:42:52.844906    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7658d9b749-k5zd5_9ae597eb-68e0-447c-bf0c-781a80684af1/kube-storage-version-migrator-operator/0.log"
Jul 23 15:42:53.765591 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:42:53.765555    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-68mqt_bc81888a-6094-4c2e-b881-6b9cd628e6c1/kube-multus/0.log"
Jul 23 15:42:53.788394 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:42:53.788370    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-q79jt_6fbaa452-6ab5-4370-ae18-fb4290ce9c10/kube-multus-additional-cni-plugins/0.log"
Jul 23 15:42:53.808997 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:42:53.808972    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-q79jt_6fbaa452-6ab5-4370-ae18-fb4290ce9c10/egress-router-binary-copy/0.log"
Jul 23 15:42:53.829943 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:42:53.829913    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-q79jt_6fbaa452-6ab5-4370-ae18-fb4290ce9c10/cni-plugins/0.log"
Jul 23 15:42:53.852879 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:42:53.852849    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-q79jt_6fbaa452-6ab5-4370-ae18-fb4290ce9c10/bond-cni-plugin/0.log"
Jul 23 15:42:53.875150 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:42:53.875124    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-q79jt_6fbaa452-6ab5-4370-ae18-fb4290ce9c10/routeoverride-cni/0.log"
Jul 23 15:42:53.895554 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:42:53.895524    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-q79jt_6fbaa452-6ab5-4370-ae18-fb4290ce9c10/whereabouts-cni-bincopy/0.log"
Jul 23 15:42:53.915170 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:42:53.915148    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-q79jt_6fbaa452-6ab5-4370-ae18-fb4290ce9c10/whereabouts-cni/0.log"
Jul 23 15:42:54.277716 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:42:54.277685    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-hvw8v_413d2464-7258-4246-85bb-ac9f03976ee5/network-metrics-daemon/0.log"
Jul 23 15:42:54.297176 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:42:54.297147    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-hvw8v_413d2464-7258-4246-85bb-ac9f03976ee5/kube-rbac-proxy/0.log"
Jul 23 15:42:55.575239 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:42:55.575198    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z4fvn_e0c66f26-06fe-4273-ba29-fa20ce89b873/ovn-controller/0.log"
Jul 23 15:42:55.593834 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:42:55.593801    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z4fvn_e0c66f26-06fe-4273-ba29-fa20ce89b873/ovn-acl-logging/1.log"
Jul 23 15:42:55.593990 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:42:55.593849    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z4fvn_e0c66f26-06fe-4273-ba29-fa20ce89b873/ovn-acl-logging/0.log"
Jul 23 15:42:55.613519 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:42:55.613491    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z4fvn_e0c66f26-06fe-4273-ba29-fa20ce89b873/kube-rbac-proxy-node/0.log"
Jul 23 15:42:55.636919 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:42:55.636895    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z4fvn_e0c66f26-06fe-4273-ba29-fa20ce89b873/kube-rbac-proxy-ovn-metrics/0.log"
Jul 23 15:42:55.654196 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:42:55.654172    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z4fvn_e0c66f26-06fe-4273-ba29-fa20ce89b873/northd/0.log"
Jul 23 15:42:55.673398 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:42:55.673367    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z4fvn_e0c66f26-06fe-4273-ba29-fa20ce89b873/nbdb/0.log"
Jul 23 15:42:55.695298 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:42:55.695266    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z4fvn_e0c66f26-06fe-4273-ba29-fa20ce89b873/sbdb/0.log"
Jul 23 15:42:55.750827 ip-10-0-141-105 kubenswrapper[2560]: I0723 15:42:55.750799    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z4fvn_e0c66f26-06fe-4273-ba29-fa20ce89b873/ovnkube-controller/0.log"