Jul 06 20:23:56.154060 ip-10-0-141-123 systemd[1]: Starting Kubernetes Kubelet...
Jul 06 20:23:56.548702 ip-10-0-141-123 kubenswrapper[2563]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 06 20:23:56.548702 ip-10-0-141-123 kubenswrapper[2563]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 06 20:23:56.548702 ip-10-0-141-123 kubenswrapper[2563]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 06 20:23:56.548702 ip-10-0-141-123 kubenswrapper[2563]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 06 20:23:56.548702 ip-10-0-141-123 kubenswrapper[2563]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 06 20:23:56.550265 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.550173    2563 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 06 20:23:56.558370 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.558337    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 06 20:23:56.558370 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.558368    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 06 20:23:56.558370 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.558372    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 06 20:23:56.558370 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.558376    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 06 20:23:56.558370 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.558380    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 06 20:23:56.558370 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.558383    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 06 20:23:56.558370 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.558386    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 06 20:23:56.558624 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.558389    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 06 20:23:56.558624 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.558393    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 06 20:23:56.558624 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.558395    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 06 20:23:56.558624 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.558398    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 06 20:23:56.558624 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.558400    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 06 20:23:56.558624 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.558403    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 06 20:23:56.558624 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.558406    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 06 20:23:56.558624 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.558408    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 06 20:23:56.558624 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.558411    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 06 20:23:56.558624 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.558414    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 06 20:23:56.558624 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.558416    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 06 20:23:56.558624 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.558419    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 06 20:23:56.558624 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.558422    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 06 20:23:56.558624 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.558424    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 06 20:23:56.558624 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.558427    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 06 20:23:56.558624 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.558429    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 06 20:23:56.558624 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.558431    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 06 20:23:56.558624 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.558434    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 06 20:23:56.558624 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.558437    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 06 20:23:56.558624 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.558440    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 06 20:23:56.559117 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.558442    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 06 20:23:56.559117 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.558445    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 06 20:23:56.559117 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.558447    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 06 20:23:56.559117 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.558450    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 06 20:23:56.559117 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.558452    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 06 20:23:56.559117 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.558462    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 06 20:23:56.559117 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.558465    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 06 20:23:56.559117 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.558467    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 06 20:23:56.559117 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.558470    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 06 20:23:56.559117 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.558472    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 06 20:23:56.559117 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.558475    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 06 20:23:56.559117 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.558478    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 06 20:23:56.559117 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.558481    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 06 20:23:56.559117 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.558484    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 06 20:23:56.559117 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.558486    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 06 20:23:56.559117 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.558489    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 06 20:23:56.559117 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.558491    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 06 20:23:56.559117 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.558494    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 06 20:23:56.559117 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.558496    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 06 20:23:56.559117 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.558499    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 06 20:23:56.559594 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.558501    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 06 20:23:56.559594 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.558504    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 06 20:23:56.559594 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.558506    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 06 20:23:56.559594 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.558510    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 06 20:23:56.559594 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.558512    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 06 20:23:56.559594 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.558515    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 06 20:23:56.559594 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.558517    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 06 20:23:56.559594 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.558522    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 06 20:23:56.559594 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.558526    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 06 20:23:56.559594 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.558530    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 06 20:23:56.559594 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.558532    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 06 20:23:56.559594 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.558535    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 06 20:23:56.559594 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.558538    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 06 20:23:56.559594 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.558540    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 06 20:23:56.559594 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.558543    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 06 20:23:56.559594 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.558545    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 06 20:23:56.559594 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.558548    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 06 20:23:56.559594 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.558550    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 06 20:23:56.559594 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.558553    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 06 20:23:56.559594 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.558555    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 06 20:23:56.560115 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.558558    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 06 20:23:56.560115 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.558561    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 06 20:23:56.560115 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.558563    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 06 20:23:56.560115 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.558566    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 06 20:23:56.560115 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.558569    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 06 20:23:56.560115 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.558571    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 06 20:23:56.560115 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.558578    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 06 20:23:56.560115 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.558581    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 06 20:23:56.560115 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.558584    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 06 20:23:56.560115 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.558587    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 06 20:23:56.560115 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.558589    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 06 20:23:56.560115 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.558592    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 06 20:23:56.560115 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.558594    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 06 20:23:56.560115 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.558597    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 06 20:23:56.560115 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.558599    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 06 20:23:56.560115 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.558604    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 06 20:23:56.560115 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.558609    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 06 20:23:56.560115 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.558612    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 06 20:23:56.560115 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.558615    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 06 20:23:56.560552 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.558617    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 06 20:23:56.560552 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.558620    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 06 20:23:56.560552 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.558623    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 06 20:23:56.560552 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.558625    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 06 20:23:56.560552 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.558628    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 06 20:23:56.560552 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.558631    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 06 20:23:56.560552 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.558633    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 06 20:23:56.560552 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.558636    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 06 20:23:56.560552 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.558639    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 06 20:23:56.560552 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.558642    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 06 20:23:56.560552 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.558645    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 06 20:23:56.560552 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.558647    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 06 20:23:56.560552 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.558651    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 06 20:23:56.560868 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.560643    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 06 20:23:56.560868 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.560651    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 06 20:23:56.560868 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.560654    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 06 20:23:56.560868 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.560658    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 06 20:23:56.560868 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.560661    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 06 20:23:56.560868 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.560664    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 06 20:23:56.560868 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.560666    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 06 20:23:56.560868 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.560669    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 06 20:23:56.560868 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.560672    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 06 20:23:56.560868 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.560675    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 06 20:23:56.560868 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.560678    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 06 20:23:56.560868 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.560680    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 06 20:23:56.560868 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.560683    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 06 20:23:56.560868 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.560685    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 06 20:23:56.560868 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.560688    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 06 20:23:56.560868 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.560691    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 06 20:23:56.560868 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.560693    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 06 20:23:56.560868 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.560696    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 06 20:23:56.560868 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.560699    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 06 20:23:56.560868 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.560702    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 06 20:23:56.561346 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.560704    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 06 20:23:56.561346 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.560707    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 06 20:23:56.561346 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.560709    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 06 20:23:56.561346 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.560712    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 06 20:23:56.561346 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.560714    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 06 20:23:56.561346 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.560717    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 06 20:23:56.561346 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.560719    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 06 20:23:56.561346 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.560722    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 06 20:23:56.561346 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.560724    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 06 20:23:56.561346 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.560742    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 06 20:23:56.561346 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.560744    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 06 20:23:56.561346 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.560747    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 06 20:23:56.561346 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.560750    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 06 20:23:56.561346 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.560753    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 06 20:23:56.561346 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.560756    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 06 20:23:56.561346 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.560759    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 06 20:23:56.561346 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.560761    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 06 20:23:56.561346 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.560764    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 06 20:23:56.561346 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.560766    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 06 20:23:56.561346 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.560769    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 06 20:23:56.561840 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.560772    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 06 20:23:56.561840 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.560774    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 06 20:23:56.561840 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.560777    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 06 20:23:56.561840 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.560780    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 06 20:23:56.561840 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.560784    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 06 20:23:56.561840 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.560788    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 06 20:23:56.561840 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.560791    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 06 20:23:56.561840 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.560794    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 06 20:23:56.561840 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.560798    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 06 20:23:56.561840 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.560801    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 06 20:23:56.561840 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.560804    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 06 20:23:56.561840 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.560807    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 06 20:23:56.561840 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.560810    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 06 20:23:56.561840 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.560812    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 06 20:23:56.561840 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.560815    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 06 20:23:56.561840 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.560817    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 06 20:23:56.561840 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.560820    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 06 20:23:56.561840 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.560823    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 06 20:23:56.561840 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.560825    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 06 20:23:56.562305 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.560828    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 06 20:23:56.562305 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.560830    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 06 20:23:56.562305 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.560834    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 06 20:23:56.562305 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.560837    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 06 20:23:56.562305 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.560840    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 06 20:23:56.562305 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.560844    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 06 20:23:56.562305 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.560847    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 06 20:23:56.562305 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.560850    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 06 20:23:56.562305 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.560853    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 06 20:23:56.562305 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.560856    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 06 20:23:56.562305 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.560859    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 06 20:23:56.562305 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.560862    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 06 20:23:56.562305 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.560865    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 06 20:23:56.562305 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.560870    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 06 20:23:56.562305 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.560874    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 06 20:23:56.562305 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.560877    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 06 20:23:56.562305 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.560879    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 06 20:23:56.562305 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.560882    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 06 20:23:56.562305 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.560885    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 06 20:23:56.562757 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.560887    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 06 20:23:56.562757 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.560890    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 06 20:23:56.562757 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.560892    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 06 20:23:56.562757 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.560895    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 06 20:23:56.562757 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.560898    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 06 20:23:56.562757 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.560901    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 06 20:23:56.562757 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.560904    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 06 20:23:56.562757 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.560906    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 06 20:23:56.562757 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.560909    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 06 20:23:56.562757 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.560912    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 06 20:23:56.562757 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.560914    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 06 20:23:56.562757 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.560917    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 06 20:23:56.562757 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.560919    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 06 20:23:56.562757 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.560922    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 06 20:23:56.562757 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.560925    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 06 20:23:56.562757 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.560927    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 06 20:23:56.562757 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.560930    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 06 20:23:56.562757 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.560933    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 06 20:23:56.562757 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.560936    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 06 20:23:56.562757 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.560939    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 06 20:23:56.563246 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.560941    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 06 20:23:56.563246 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.561339    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 06 20:23:56.563246 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.561344    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 06 20:23:56.563246 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.561347    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 06 20:23:56.563246 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.561350    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 06 20:23:56.563246 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.561353    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 06 20:23:56.563246 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.561356    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 06 20:23:56.563246 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.561359    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 06 20:23:56.563246 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.561362    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 06 20:23:56.563246 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.561365    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 06 20:23:56.563246 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.561368    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 06 20:23:56.563246 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.561371    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 06 20:23:56.563246 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.561375    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 06 20:23:56.563246 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.561379    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 06 20:23:56.563246 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.561382    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 06 20:23:56.563246 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.561385    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 06 20:23:56.563246 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.561387    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 06 20:23:56.563246 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.561391    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 06 20:23:56.563246 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.561394    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 06 20:23:56.563697 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.561397    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 06 20:23:56.563697 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.561400    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 06 20:23:56.563697 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.561402    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 06 20:23:56.563697 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.561405    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 06 20:23:56.563697 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.561408    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 06 20:23:56.563697 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.561410    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 06 20:23:56.563697 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.561412    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 06 20:23:56.563697 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.561415    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 06 20:23:56.563697 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.561417    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 06 20:23:56.563697 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.561420    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 06 20:23:56.563697 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.561422    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 06 20:23:56.563697 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.561425    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 06 20:23:56.563697 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.561428    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 06 20:23:56.563697 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.561430    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 06 20:23:56.563697 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.561433    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 06 20:23:56.563697 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.561435    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 06 20:23:56.563697 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.561438    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 06 20:23:56.563697 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.561441    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 06 20:23:56.563697 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.561444    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 06 20:23:56.563697 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.561447    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 06 20:23:56.564185 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.561450    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 06 20:23:56.564185 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.561453    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 06 20:23:56.564185 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.561455    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 06 20:23:56.564185 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.561458    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 06 20:23:56.564185 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.561460    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 06 20:23:56.564185 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.561463    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 06 20:23:56.564185 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.561465    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 06 20:23:56.564185 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.561469    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 06 20:23:56.564185 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.561472    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 06 20:23:56.564185 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.561475    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 06 20:23:56.564185 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.561478    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 06 20:23:56.564185 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.561484    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 06 20:23:56.564185 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.561487    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 06 20:23:56.564185 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.561490    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 06 20:23:56.564185 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.561493    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 06 20:23:56.564185 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.561495    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 06 20:23:56.564185 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.561498    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 06 20:23:56.564185 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.561500    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 06 20:23:56.564185 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.561502    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 06 20:23:56.564185 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.561505    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 06 20:23:56.564687 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.561507    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 06 20:23:56.564687 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.561510    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 06 20:23:56.564687 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.561512    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 06 20:23:56.564687 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.561517    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 06 20:23:56.564687 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.561520    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 06 20:23:56.564687 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.561523    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 06 20:23:56.564687 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.561525    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 06 20:23:56.564687 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.561528    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 06 20:23:56.564687 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.561531    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 06 20:23:56.564687 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.561533    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 06 20:23:56.564687 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.561536    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 06 20:23:56.564687 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.561538    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 06 20:23:56.564687 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.561541    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 06 20:23:56.564687 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.561544    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 06 20:23:56.564687 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.561546    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 06 20:23:56.564687 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.561548    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 06 20:23:56.564687 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.561551    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 06 20:23:56.564687 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.561553    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 06 20:23:56.564687 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.561556    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 06 20:23:56.564687 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.561558    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 06 20:23:56.565174 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.561561    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 06 20:23:56.565174 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.561563    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 06 20:23:56.565174 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.561566    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 06 20:23:56.565174 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.561568    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 06 20:23:56.565174 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.561571    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 06 20:23:56.565174 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.561574    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 06 20:23:56.565174 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.561577    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 06 20:23:56.565174 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.561579    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 06 20:23:56.565174 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.561581    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 06 20:23:56.565174 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.561584    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 06 20:23:56.565174 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.561586    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 06 20:23:56.565174 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.561589    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 06 20:23:56.565174 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.561591    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 06 20:23:56.565174 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.561594    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 06 20:23:56.565174 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.561596    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 06 20:23:56.565174 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.561602    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 06 20:23:56.565174 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.561605    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 06 20:23:56.565174 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.561607    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 06 20:23:56.565174 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.561610    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 06 20:23:56.565626 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.561612    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 06 20:23:56.565626 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.561615    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 06 20:23:56.565626 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.561687    2563 flags.go:64] FLAG: --address="0.0.0.0"
Jul 06 20:23:56.565626 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.561695    2563 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 06 20:23:56.565626 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.561701    2563 flags.go:64] FLAG: --anonymous-auth="true"
Jul 06 20:23:56.565626 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.561706    2563 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 06 20:23:56.565626 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.561710    2563 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 06 20:23:56.565626 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.561713    2563 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 06 20:23:56.565626 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.561717    2563 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 06 20:23:56.565626 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.561721    2563 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 06 20:23:56.565626 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.561724    2563 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 06 20:23:56.565626 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.561741    2563 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 06 20:23:56.565626 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.561744    2563 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 06 20:23:56.565626 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.561747    2563 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 06 20:23:56.565626 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.561750    2563 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 06 20:23:56.565626 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.561754    2563 flags.go:64] FLAG: --cgroup-root=""
Jul 06 20:23:56.565626 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.561757    2563 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 06 20:23:56.565626 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.561760    2563 flags.go:64] FLAG: --client-ca-file=""
Jul 06 20:23:56.565626 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.561763    2563 flags.go:64] FLAG: --cloud-provider="external"
Jul 06 20:23:56.565626 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.561766    2563 flags.go:64] FLAG: --cluster-dns="[]"
Jul 06 20:23:56.565626 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.561771    2563 flags.go:64] FLAG: --cluster-domain=""
Jul 06 20:23:56.565626 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.561774    2563 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 06 20:23:56.565626 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.561777    2563 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 06 20:23:56.565626 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.561780    2563 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 06 20:23:56.566207 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.561783    2563 flags.go:64] FLAG: --container-log-max-files="5"
Jul 06 20:23:56.566207 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.561787    2563 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 06 20:23:56.566207 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.561790    2563 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 06 20:23:56.566207 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.561793    2563 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 06 20:23:56.566207 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.561798    2563 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 06 20:23:56.566207 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.561801    2563 flags.go:64] FLAG: --contention-profiling="false"
Jul 06 20:23:56.566207 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.561804    2563 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 06 20:23:56.566207 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.561807    2563 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 06 20:23:56.566207 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.561810    2563 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 06 20:23:56.566207 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.561813    2563 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 06 20:23:56.566207 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.561819    2563 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 06 20:23:56.566207 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.561822    2563 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 06 20:23:56.566207 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.561825    2563 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 06 20:23:56.566207 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.561828    2563 flags.go:64] FLAG: --enable-load-reader="false"
Jul 06 20:23:56.566207 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.561831    2563 flags.go:64] FLAG: --enable-server="true"
Jul 06 20:23:56.566207 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.561834    2563 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 06 20:23:56.566207 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.561838    2563 flags.go:64] FLAG: --event-burst="100"
Jul 06 20:23:56.566207 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.561842    2563 flags.go:64] FLAG: --event-qps="50"
Jul 06 20:23:56.566207 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.561845    2563 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 06 20:23:56.566207 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.561848    2563 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 06 20:23:56.566207 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.561851    2563 flags.go:64] FLAG: --eviction-hard=""
Jul 06 20:23:56.566207 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.561855    2563 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 06 20:23:56.566207 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.561858    2563 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 06 20:23:56.566207 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.561861    2563 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 06 20:23:56.566207 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.561864    2563 flags.go:64] FLAG: --eviction-soft=""
Jul 06 20:23:56.566815 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.561867    2563 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 06 20:23:56.566815 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.561870    2563 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 06 20:23:56.566815 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.561873    2563 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 06 20:23:56.566815 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.561877    2563 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 06 20:23:56.566815 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.561879    2563 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 06 20:23:56.566815 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.561882    2563 flags.go:64] FLAG: --fail-swap-on="true"
Jul 06 20:23:56.566815 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.561885    2563 flags.go:64] FLAG: --feature-gates=""
Jul 06 20:23:56.566815 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.561889    2563 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 06 20:23:56.566815 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.561892    2563 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 06 20:23:56.566815 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.561895    2563 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 06 20:23:56.566815 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.561898    2563 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 06 20:23:56.566815 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.561902    2563 flags.go:64] FLAG: --healthz-port="10248"
Jul 06 20:23:56.566815 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.561906    2563 flags.go:64] FLAG: --help="false"
Jul 06 20:23:56.566815 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.561908    2563 flags.go:64] FLAG: --hostname-override="ip-10-0-141-123.ec2.internal"
Jul 06 20:23:56.566815 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.561912    2563 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 06 20:23:56.566815 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.561914    2563 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 06 20:23:56.566815 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.561917    2563 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 06 20:23:56.566815 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.561922    2563 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 06 20:23:56.566815 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.561925    2563 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 06 20:23:56.566815 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.561928    2563 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 06 20:23:56.566815 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.561931    2563 flags.go:64] FLAG: --image-service-endpoint=""
Jul 06 20:23:56.566815 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.561934    2563 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 06 20:23:56.566815 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.561937    2563 flags.go:64] FLAG: --kube-api-burst="100"
Jul 06 20:23:56.567390 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.561940    2563 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 06 20:23:56.567390 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.561943    2563 flags.go:64] FLAG: --kube-api-qps="50"
Jul 06 20:23:56.567390 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.561946    2563 flags.go:64] FLAG: --kube-reserved=""
Jul 06 20:23:56.567390 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.561949    2563 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 06 20:23:56.567390 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.561952    2563 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 06 20:23:56.567390 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.561955    2563 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 06 20:23:56.567390 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.561957    2563 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 06 20:23:56.567390 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.561960    2563 flags.go:64] FLAG: --lock-file=""
Jul 06 20:23:56.567390 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.561963    2563 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 06 20:23:56.567390 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.561966    2563 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 06 20:23:56.567390 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.561969    2563 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 06 20:23:56.567390 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.561974    2563 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 06 20:23:56.567390 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.561978    2563 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 06 20:23:56.567390 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.561981    2563 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 06 20:23:56.567390 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.561984    2563 flags.go:64] FLAG: --logging-format="text"
Jul 06 20:23:56.567390 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.561986    2563 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 06 20:23:56.567390 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.561990    2563 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 06 20:23:56.567390 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.561992    2563 flags.go:64] FLAG: --manifest-url=""
Jul 06 20:23:56.567390 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.561995    2563 flags.go:64] FLAG: --manifest-url-header=""
Jul 06 20:23:56.567390 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.561999    2563 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 06 20:23:56.567390 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.562004    2563 flags.go:64] FLAG: --max-open-files="1000000"
Jul 06 20:23:56.567390 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.562007    2563 flags.go:64] FLAG: --max-pods="110"
Jul 06 20:23:56.567390 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.562011    2563 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 06 20:23:56.567390 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.562014    2563 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 06 20:23:56.567390 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.562017    2563 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 06 20:23:56.568043 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.562020    2563 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 06 20:23:56.568043 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.562024    2563 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 06 20:23:56.568043 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.562027    2563 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 06 20:23:56.568043 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.562030    2563 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 06 20:23:56.568043 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.562037    2563 flags.go:64] FLAG: --node-status-max-images="50"
Jul 06 20:23:56.568043 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.562040    2563 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 06 20:23:56.568043 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.562043    2563 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 06 20:23:56.568043 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.562046    2563 flags.go:64] FLAG: --pod-cidr=""
Jul 06 20:23:56.568043 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.562049    2563 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5"
Jul 06 20:23:56.568043 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.562054    2563 flags.go:64] FLAG: --pod-manifest-path=""
Jul 06 20:23:56.568043 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.562057    2563 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 06 20:23:56.568043 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.562060    2563 flags.go:64] FLAG: --pods-per-core="0"
Jul 06 20:23:56.568043 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.562063    2563 flags.go:64] FLAG: --port="10250"
Jul 06 20:23:56.568043 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.562066    2563 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 06 20:23:56.568043 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.562069    2563 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0178d84ab0205128f"
Jul 06 20:23:56.568043 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.562072    2563 flags.go:64] FLAG: --qos-reserved=""
Jul 06 20:23:56.568043 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.562075    2563 flags.go:64] FLAG: --read-only-port="10255"
Jul 06 20:23:56.568043 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.562078    2563 flags.go:64] FLAG: --register-node="true"
Jul 06 20:23:56.568043 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.562081    2563 flags.go:64] FLAG: --register-with-taints=""
Jul 06 20:23:56.568043 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.562085    2563 flags.go:64] FLAG: --registry-burst="10"
Jul 06 20:23:56.568043 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.562088    2563 flags.go:64] FLAG: --registry-qps="5"
Jul 06 20:23:56.568043 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.562091    2563 flags.go:64] FLAG: --reserved-cpus=""
Jul 06 20:23:56.568043 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.562094    2563 flags.go:64] FLAG: --reserved-memory=""
Jul 06 20:23:56.568043 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.562097    2563 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 06 20:23:56.568043 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.562100    2563 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 06 20:23:56.568647 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.562103    2563 flags.go:64] FLAG: --rotate-certificates="false"
Jul 06 20:23:56.568647 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.562106    2563 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 06 20:23:56.568647 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.562108    2563 flags.go:64] FLAG: --runonce="false"
Jul 06 20:23:56.568647 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.562113    2563 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 06 20:23:56.568647 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.562116    2563 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 06 20:23:56.568647 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.562119    2563 flags.go:64] FLAG: --seccomp-default="false"
Jul 06 20:23:56.568647 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.562122    2563 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 06 20:23:56.568647 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.562125    2563 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 06 20:23:56.568647 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.562129    2563 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 06 20:23:56.568647 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.562132    2563 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 06 20:23:56.568647 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.562135    2563 flags.go:64] FLAG: --storage-driver-password="root"
Jul 06 20:23:56.568647 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.562138    2563 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 06 20:23:56.568647 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.562141    2563 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 06 20:23:56.568647 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.562144    2563 flags.go:64] FLAG: --storage-driver-user="root"
Jul 06 20:23:56.568647 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.562147    2563 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 06 20:23:56.568647 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.562150    2563 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 06 20:23:56.568647 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.562153    2563 flags.go:64] FLAG: --system-cgroups=""
Jul 06 20:23:56.568647 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.562155    2563 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 06 20:23:56.568647 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.562161    2563 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 06 20:23:56.568647 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.562164    2563 flags.go:64] FLAG: --tls-cert-file=""
Jul 06 20:23:56.568647 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.562166    2563 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 06 20:23:56.568647 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.562170    2563 flags.go:64] FLAG: --tls-min-version=""
Jul 06 20:23:56.568647 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.562173    2563 flags.go:64] FLAG: --tls-private-key-file=""
Jul 06 20:23:56.568647 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.562175    2563 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 06 20:23:56.568647 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.562179    2563 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 06 20:23:56.569698 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.562181    2563 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 06 20:23:56.569698 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.562184    2563 flags.go:64] FLAG: --v="2"
Jul 06 20:23:56.569698 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.562188    2563 flags.go:64] FLAG: --version="false"
Jul 06 20:23:56.569698 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.562192    2563 flags.go:64] FLAG: --vmodule=""
Jul 06 20:23:56.569698 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.562196    2563 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 06 20:23:56.569698 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.562199    2563 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 06 20:23:56.569698 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.562308    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 06 20:23:56.569698 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.562312    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 06 20:23:56.569698 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.562314    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 06 20:23:56.569698 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.562317    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 06 20:23:56.569698 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.562321    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 06 20:23:56.569698 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.562323    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 06 20:23:56.569698 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.562326    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 06 20:23:56.569698 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.562328    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 06 20:23:56.569698 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.562331    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 06 20:23:56.569698 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.562335    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 06 20:23:56.569698 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.562338    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 06 20:23:56.569698 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.562340    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 06 20:23:56.569698 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.562342    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 06 20:23:56.569698 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.562345    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 06 20:23:56.569698 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.562349    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 06 20:23:56.570604 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.562353    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 06 20:23:56.570604 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.562356    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 06 20:23:56.570604 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.562359    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 06 20:23:56.570604 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.562362    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 06 20:23:56.570604 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.562364    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 06 20:23:56.570604 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.562367    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 06 20:23:56.570604 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.562369    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 06 20:23:56.570604 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.562372    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 06 20:23:56.570604 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.562374    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 06 20:23:56.570604 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.562377    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 06 20:23:56.570604 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.562380    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 06 20:23:56.570604 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.562382    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 06 20:23:56.570604 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.562385    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 06 20:23:56.570604 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.562387    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 06 20:23:56.570604 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.562390    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 06 20:23:56.570604 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.562393    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 06 20:23:56.570604 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.562395    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 06 20:23:56.570604 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.562398    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 06 20:23:56.570604 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.562400    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 06 20:23:56.571324 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.562403    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 06 20:23:56.571324 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.562405    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 06 20:23:56.571324 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.562408    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 06 20:23:56.571324 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.562411    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 06 20:23:56.571324 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.562413    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 06 20:23:56.571324 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.562415    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 06 20:23:56.571324 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.562418    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 06 20:23:56.571324 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.562421    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 06 20:23:56.571324 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.562424    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 06 20:23:56.571324 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.562426    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 06 20:23:56.571324 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.562429    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 06 20:23:56.571324 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.562431    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 06 20:23:56.571324 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.562434    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 06 20:23:56.571324 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.562436    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 06 20:23:56.571324 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.562439    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 06 20:23:56.571324 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.562441    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 06 20:23:56.571324 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.562444    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 06 20:23:56.571324 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.562446    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 06 20:23:56.571324 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.562449    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 06 20:23:56.571324 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.562451    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 06 20:23:56.571815 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.562454    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 06 20:23:56.571815 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.562456    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 06 20:23:56.571815 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.562459    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 06 20:23:56.571815 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.562461    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 06 20:23:56.571815 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.562464    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 06 20:23:56.571815 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.562466    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 06 20:23:56.571815 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.562469    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 06 20:23:56.571815 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.562471    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 06 20:23:56.571815 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.562474    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 06 20:23:56.571815 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.562477    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 06 20:23:56.571815 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.562479    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 06 20:23:56.571815 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.562481    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 06 20:23:56.571815 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.562484    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 06 20:23:56.571815 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.562487    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 06 20:23:56.571815 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.562489    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 06 20:23:56.571815 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.562492    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 06 20:23:56.571815 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.562494    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 06 20:23:56.571815 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.562497    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 06 20:23:56.571815 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.562499    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 06 20:23:56.571815 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.562503    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 06 20:23:56.572313 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.562505    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 06 20:23:56.572313 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.562508    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 06 20:23:56.572313 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.562511    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 06 20:23:56.572313 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.562513    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 06 20:23:56.572313 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.562516    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 06 20:23:56.572313 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.562518    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 06 20:23:56.572313 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.562520    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 06 20:23:56.572313 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.562523    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 06 20:23:56.572313 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.562527    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 06 20:23:56.572313 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.562530    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 06 20:23:56.572313 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.562534    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 06 20:23:56.572313 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.562537    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 06 20:23:56.572313 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.562540    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 06 20:23:56.572313 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.562543    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 06 20:23:56.572313 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.562546    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 06 20:23:56.572313 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.562549    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 06 20:23:56.572313 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.562551    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 06 20:23:56.572313 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.562553    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 06 20:23:56.572313 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.562556    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 06 20:23:56.572835 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.562558    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 06 20:23:56.572835 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.562561    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 06 20:23:56.572835 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.562563    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 06 20:23:56.572835 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.562566    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 06 20:23:56.572835 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.562569    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 06 20:23:56.572835 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.562571    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 06 20:23:56.572835 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.563103    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 06 20:23:56.572835 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.571098    2563 server.go:529] "Kubelet version" kubeletVersion="v1.34.8"
Jul 06 20:23:56.572835 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.571120    2563 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 06 20:23:56.572835 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.572712    2563 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 06 20:23:56.572835 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.572742    2563 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 06 20:23:56.572835 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.572839    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 06 20:23:56.573369 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.572851    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 06 20:23:56.573369 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.572857    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 06 20:23:56.573369 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.572862    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 06 20:23:56.573369 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.572867    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 06 20:23:56.573369 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.572872    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 06 20:23:56.573369 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.572877    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 06 20:23:56.573369 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.572881    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 06 20:23:56.573369 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.572885    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 06 20:23:56.573369 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.572889    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 06 20:23:56.573369 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.572894    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 06 20:23:56.573369 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.572898    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 06 20:23:56.573369 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.572902    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 06 20:23:56.573369 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.572907    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 06 20:23:56.573369 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.572911    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 06 20:23:56.573369 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.572915    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 06 20:23:56.573369 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.572919    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 06 20:23:56.573369 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.572923    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 06 20:23:56.573369 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.572927    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 06 20:23:56.573369 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.572932    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 06 20:23:56.574252 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.572936    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 06 20:23:56.574252 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.572940    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 06 20:23:56.574252 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.572944    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 06 20:23:56.574252 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.572949    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 06 20:23:56.574252 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.572953    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 06 20:23:56.574252 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.572957    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 06 20:23:56.574252 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.572961    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 06 20:23:56.574252 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.572966    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 06 20:23:56.574252 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.572970    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 06 20:23:56.574252 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.572975    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 06 20:23:56.574252 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.572979    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 06 20:23:56.574252 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.572983    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 06 20:23:56.574252 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.572988    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 06 20:23:56.574252 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.572992    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 06 20:23:56.574252 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.572997    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 06 20:23:56.574252 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.573002    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 06 20:23:56.574252 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.573006    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 06 20:23:56.574252 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.573010    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 06 20:23:56.574252 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.573015    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 06 20:23:56.575177 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.573018    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 06 20:23:56.575177 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.573022    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 06 20:23:56.575177 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.573027    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 06 20:23:56.575177 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.573031    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 06 20:23:56.575177 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.573036    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 06 20:23:56.575177 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.573040    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 06 20:23:56.575177 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.573044    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 06 20:23:56.575177 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.573048    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 06 20:23:56.575177 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.573053    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 06 20:23:56.575177 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.573057    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 06 20:23:56.575177 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.573061    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 06 20:23:56.575177 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.573066    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 06 20:23:56.575177 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.573070    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 06 20:23:56.575177 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.573074    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 06 20:23:56.575177 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.573079    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 06 20:23:56.575177 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.573083    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 06 20:23:56.575177 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.573087    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 06 20:23:56.575177 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.573091    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 06 20:23:56.575177 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.573095    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 06 20:23:56.575177 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.573100    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 06 20:23:56.576286 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.573104    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 06 20:23:56.576286 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.573108    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 06 20:23:56.576286 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.573113    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 06 20:23:56.576286 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.573118    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 06 20:23:56.576286 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.573122    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 06 20:23:56.576286 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.573127    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 06 20:23:56.576286 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.573131    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 06 20:23:56.576286 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.573135    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 06 20:23:56.576286 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.573141    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 06 20:23:56.576286 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.573147    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 06 20:23:56.576286 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.573151    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 06 20:23:56.576286 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.573155    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 06 20:23:56.576286 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.573159    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 06 20:23:56.576286 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.573164    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 06 20:23:56.576286 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.573168    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 06 20:23:56.576286 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.573174    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 06 20:23:56.576286 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.573179    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 06 20:23:56.576286 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.573185    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 06 20:23:56.576286 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.573190    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 06 20:23:56.576286 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.573196    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 06 20:23:56.576991 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.573200    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 06 20:23:56.576991 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.573205    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 06 20:23:56.576991 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.573209    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 06 20:23:56.576991 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.573213    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 06 20:23:56.576991 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.573217    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 06 20:23:56.576991 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.573222    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 06 20:23:56.576991 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.573226    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 06 20:23:56.576991 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.573230    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 06 20:23:56.576991 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.573234    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 06 20:23:56.576991 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.573238    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 06 20:23:56.576991 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.573244    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 06 20:23:56.576991 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.573252    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 06 20:23:56.576991 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.573258    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 06 20:23:56.576991 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.573262    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 06 20:23:56.576991 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.573267    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 06 20:23:56.576991 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.573272    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 06 20:23:56.576991 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.573286    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 06 20:23:56.576991 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.573290    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 06 20:23:56.576991 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.573294    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 06 20:23:56.577621 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.573300    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 06 20:23:56.577621 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.573308    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 06 20:23:56.577621 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.573568    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 06 20:23:56.577621 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.573577    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 06 20:23:56.577621 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.573582    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 06 20:23:56.577621 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.573586    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 06 20:23:56.577621 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.573591    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 06 20:23:56.577621 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.573596    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 06 20:23:56.577621 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.573600    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 06 20:23:56.577621 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.573605    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 06 20:23:56.577621 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.573609    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 06 20:23:56.577621 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.573614    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 06 20:23:56.577621 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.573619    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 06 20:23:56.577621 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.573623    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 06 20:23:56.577621 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.573628    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 06 20:23:56.578155 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.573633    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 06 20:23:56.578155 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.573637    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 06 20:23:56.578155 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.573641    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 06 20:23:56.578155 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.573646    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 06 20:23:56.578155 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.573650    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 06 20:23:56.578155 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.573655    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 06 20:23:56.578155 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.573659    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 06 20:23:56.578155 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.573664    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 06 20:23:56.578155 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.573668    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 06 20:23:56.578155 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.573672    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 06 20:23:56.578155 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.573678    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 06 20:23:56.578155 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.573682    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 06 20:23:56.578155 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.573686    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 06 20:23:56.578155 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.573691    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 06 20:23:56.578155 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.573695    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 06 20:23:56.578155 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.573708    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 06 20:23:56.578155 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.573713    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 06 20:23:56.578155 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.573718    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 06 20:23:56.578155 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.573723    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 06 20:23:56.578155 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.573729    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 06 20:23:56.578686 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.573735    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 06 20:23:56.578686 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.573740    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 06 20:23:56.578686 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.573745    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 06 20:23:56.578686 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.573750    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 06 20:23:56.578686 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.573772    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 06 20:23:56.578686 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.573777    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 06 20:23:56.578686 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.573782    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 06 20:23:56.578686 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.573787    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 06 20:23:56.578686 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.573791    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 06 20:23:56.578686 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.573796    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 06 20:23:56.578686 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.573800    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 06 20:23:56.578686 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.573804    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 06 20:23:56.578686 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.573808    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 06 20:23:56.578686 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.573813    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 06 20:23:56.578686 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.573818    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 06 20:23:56.578686 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.573823    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 06 20:23:56.578686 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.573827    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 06 20:23:56.578686 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.573832    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 06 20:23:56.578686 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.573836    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 06 20:23:56.578686 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.573841    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 06 20:23:56.579272 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.573845    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 06 20:23:56.579272 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.573851    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 06 20:23:56.579272 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.573858    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 06 20:23:56.579272 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.573864    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 06 20:23:56.579272 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.573868    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 06 20:23:56.579272 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.573873    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 06 20:23:56.579272 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.573878    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 06 20:23:56.579272 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.573884    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 06 20:23:56.579272 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.573901    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 06 20:23:56.579272 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.573907    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 06 20:23:56.579272 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.573913    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 06 20:23:56.579272 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.573918    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 06 20:23:56.579272 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.573922    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 06 20:23:56.579272 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.573928    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 06 20:23:56.579272 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.573932    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 06 20:23:56.579272 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.573937    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 06 20:23:56.579272 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.573942    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 06 20:23:56.579272 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.573946    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 06 20:23:56.579272 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.573951    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 06 20:23:56.579796 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.573956    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 06 20:23:56.579796 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.573960    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 06 20:23:56.579796 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.573964    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 06 20:23:56.579796 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.573969    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 06 20:23:56.579796 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.573974    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 06 20:23:56.579796 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.573978    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 06 20:23:56.579796 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.573983    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 06 20:23:56.579796 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.573988    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 06 20:23:56.579796 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.573992    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 06 20:23:56.579796 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.573996    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 06 20:23:56.579796 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.574001    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 06 20:23:56.579796 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.574006    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 06 20:23:56.579796 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.574010    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 06 20:23:56.579796 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.574014    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 06 20:23:56.579796 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.574019    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 06 20:23:56.579796 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.574023    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 06 20:23:56.579796 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.574028    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 06 20:23:56.579796 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.574032    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 06 20:23:56.579796 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.574037    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 06 20:23:56.580356 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.574041    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 06 20:23:56.580356 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.574046    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 06 20:23:56.580356 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.574050    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 06 20:23:56.580356 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.574064    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 06 20:23:56.580356 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.574069    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 06 20:23:56.580356 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.574073    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 06 20:23:56.580356 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.574077    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 06 20:23:56.580356 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:56.574082    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 06 20:23:56.580356 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.574090    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 06 20:23:56.580356 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.574280    2563 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 06 20:23:56.580356 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.577335    2563 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 06 20:23:56.580356 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.579012    2563 server.go:1019] "Starting client certificate rotation"
Jul 06 20:23:56.580356 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.579122    2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 06 20:23:56.580356 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.579166    2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 06 20:23:56.583508 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.583491    2563 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 06 20:23:56.585038 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.585016    2563 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 06 20:23:56.598086 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.598064    2563 log.go:25] "Validated CRI v1 runtime API"
Jul 06 20:23:56.600271 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.600256    2563 log.go:25] "Validated CRI v1 image API"
Jul 06 20:23:56.601097 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.601071    2563 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 06 20:23:56.604685 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.604663    2563 fs.go:135] Filesystem UUIDs: map[54db3f02-fa9e-4f47-8606-08dfe5d4c125:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 cf51711d-1cf8-4057-bd1a-86ad1215b2fd:/dev/nvme0n1p3]
Jul 06 20:23:56.604763 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.604684    2563 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 06 20:23:56.609854 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.609835    2563 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 06 20:23:56.610023 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.609900    2563 manager.go:216] Machine: {Timestamp:2026-07-06 20:23:56.608485808 +0000 UTC m=+0.324596462 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3130448 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec24243962477fb320e8e5b8023fe08c SystemUUID:ec242439-6247-7fb3-20e8-e5b8023fe08c BootID:fe9b074c-69eb-44b2-97bc-53f5b44c1d54 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:76:35:2c:66:cb Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:76:35:2c:66:cb Speed:0 Mtu:9001} {Name:ovs-system MacAddress:b6:cd:74:0e:a2:11 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 06 20:23:56.610023 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.610024    2563 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 06 20:23:56.610123 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.610105    2563 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 06 20:23:56.613426 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.613402    2563 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 06 20:23:56.613582 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.613430    2563 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-123.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 06 20:23:56.615143 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.615132    2563 topology_manager.go:138] "Creating topology manager with none policy"
Jul 06 20:23:56.615178 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.615146    2563 container_manager_linux.go:309] "Creating device plugin manager"
Jul 06 20:23:56.615178 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.615154    2563 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 06 20:23:56.615178 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.615167    2563 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 06 20:23:56.616817 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.616797    2563 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 06 20:23:56.619951 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.619939    2563 state_mem.go:36] "Initialized new in-memory state store"
Jul 06 20:23:56.620067 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.620057    2563 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 06 20:23:56.620099 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.620089    2563 kubelet.go:486] "Attempting to sync node with API server"
Jul 06 20:23:56.620128 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.620108    2563 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 06 20:23:56.620177 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.620129    2563 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 06 20:23:56.620177 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.620138    2563 kubelet.go:398] "Adding apiserver pod source"
Jul 06 20:23:56.620177 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.620148    2563 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 06 20:23:56.620577 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.620565    2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 06 20:23:56.620616 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.620599    2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 06 20:23:56.622813 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.622794    2563 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1"
Jul 06 20:23:56.624816 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.624788    2563 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 06 20:23:56.624893 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.624839    2563 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 06 20:23:56.626044 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.626026    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 06 20:23:56.626120 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.626050    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 06 20:23:56.626120 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.626057    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 06 20:23:56.626120 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.626063    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 06 20:23:56.626120 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.626072    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 06 20:23:56.626120 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.626077    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 06 20:23:56.626120 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.626085    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 06 20:23:56.626120 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.626091    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 06 20:23:56.626120 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.626099    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 06 20:23:56.626120 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.626104    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 06 20:23:56.626120 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.626114    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 06 20:23:56.626120 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.626119    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 06 20:23:56.627473 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.627457    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 06 20:23:56.627473 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.627471    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 06 20:23:56.631028 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.631013    2563 server.go:1268] "Started kubelet"
Jul 06 20:23:56.631106 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.631049    2563 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-141-123.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 06 20:23:56.631177 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:23:56.631099    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-141-123.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 06 20:23:56.631177 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.631101    2563 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 06 20:23:56.631177 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.631155    2563 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 06 20:23:56.631300 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.631196    2563 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 06 20:23:56.631406 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:23:56.631116    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 06 20:23:56.631906 ip-10-0-141-123 systemd[1]: Started Kubernetes Kubelet.
Jul 06 20:23:56.633193 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.633176    2563 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 06 20:23:56.633419 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.633404    2563 server.go:310] "Adding debug handlers to kubelet server"
Jul 06 20:23:56.637249 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:23:56.637225    2563 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 06 20:23:56.639989 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.639970    2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 06 20:23:56.640066 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.639974    2563 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 06 20:23:56.640374 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:23:56.640353    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-123.ec2.internal\" not found"
Jul 06 20:23:56.640710 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.640690    2563 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 06 20:23:56.640710 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.640713    2563 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 06 20:23:56.640914 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.640760    2563 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 06 20:23:56.640914 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.640832    2563 factory.go:55] Registering systemd factory
Jul 06 20:23:56.640914 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.640872    2563 reconstruct.go:97] "Volume reconstruction finished"
Jul 06 20:23:56.640914 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.640882    2563 reconciler.go:29] "Reconciler: start to sync state"
Jul 06 20:23:56.640914 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.640872    2563 factory.go:223] Registration of the systemd container factory successfully
Jul 06 20:23:56.641639 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:23:56.640376    2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-123.ec2.internal.18bfcd0dcdadc70d  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-123.ec2.internal,UID:ip-10-0-141-123.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-141-123.ec2.internal,},FirstTimestamp:2026-07-06 20:23:56.630992653 +0000 UTC m=+0.347103307,LastTimestamp:2026-07-06 20:23:56.630992653 +0000 UTC m=+0.347103307,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-123.ec2.internal,}"
Jul 06 20:23:56.642211 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.642196    2563 factory.go:153] Registering CRI-O factory
Jul 06 20:23:56.642289 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.642245    2563 factory.go:223] Registration of the crio container factory successfully
Jul 06 20:23:56.642346 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.642313    2563 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 06 20:23:56.642346 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.642339    2563 factory.go:103] Registering Raw factory
Jul 06 20:23:56.642445 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.642362    2563 manager.go:1196] Started watching for new ooms in manager
Jul 06 20:23:56.643004 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.642987    2563 manager.go:349] Starting recovery of all containers
Jul 06 20:23:56.643547 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:23:56.643518    2563 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-141-123.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 06 20:23:56.644054 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:23:56.644026    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 06 20:23:56.649267 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.649242    2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-6st6l"
Jul 06 20:23:56.653964 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.653852    2563 manager.go:354] Recovery completed
Jul 06 20:23:56.657703 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.657685    2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-6st6l"
Jul 06 20:23:56.658296 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.658280    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 06 20:23:56.661332 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.661315    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-123.ec2.internal" event="NodeHasSufficientMemory"
Jul 06 20:23:56.661400 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.661346    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-123.ec2.internal" event="NodeHasNoDiskPressure"
Jul 06 20:23:56.661400 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.661357    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-123.ec2.internal" event="NodeHasSufficientPID"
Jul 06 20:23:56.663153 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.663135    2563 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 06 20:23:56.663153 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.663152    2563 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 06 20:23:56.663290 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.663165    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 06 20:23:56.663290 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.663171    2563 state_mem.go:36] "Initialized new in-memory state store"
Jul 06 20:23:56.663598 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:23:56.663528    2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-123.ec2.internal.18bfcd0dcf7cb91a  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-123.ec2.internal,UID:ip-10-0-141-123.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-141-123.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-141-123.ec2.internal,},FirstTimestamp:2026-07-06 20:23:56.66133225 +0000 UTC m=+0.377442911,LastTimestamp:2026-07-06 20:23:56.66133225 +0000 UTC m=+0.377442911,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-123.ec2.internal,}"
Jul 06 20:23:56.664555 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.664535    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-123.ec2.internal" event="NodeHasSufficientMemory"
Jul 06 20:23:56.664622 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.664571    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-123.ec2.internal" event="NodeHasNoDiskPressure"
Jul 06 20:23:56.664622 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.664585    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-123.ec2.internal" event="NodeHasSufficientPID"
Jul 06 20:23:56.664622 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.664586    2563 policy_none.go:49] "None policy: Start"
Jul 06 20:23:56.664622 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.664610    2563 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 06 20:23:56.664622 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.664621    2563 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 06 20:23:56.665722 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.665710    2563 policy_none.go:47] "Start"
Jul 06 20:23:56.701432 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.701413    2563 manager.go:337] "Starting Device Plugin manager"
Jul 06 20:23:56.701523 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:23:56.701448    2563 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 06 20:23:56.701523 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.701458    2563 server.go:87] "Starting device plugin registration server"
Jul 06 20:23:56.701728 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.701714    2563 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 06 20:23:56.701793 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.701732    2563 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 06 20:23:56.701905 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.701887    2563 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 06 20:23:56.701994 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.701972    2563 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 06 20:23:56.702088 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.702067    2563 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 06 20:23:56.703588 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:23:56.703565    2563 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 06 20:23:56.703670 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:23:56.703645    2563 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-123.ec2.internal\" not found"
Jul 06 20:23:56.803129 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.803031    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 06 20:23:56.804555 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.804138    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-123.ec2.internal" event="NodeHasSufficientMemory"
Jul 06 20:23:56.804555 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.804167    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-123.ec2.internal" event="NodeHasNoDiskPressure"
Jul 06 20:23:56.804555 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.804178    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-123.ec2.internal" event="NodeHasSufficientPID"
Jul 06 20:23:56.804555 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.804207    2563 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-141-123.ec2.internal"
Jul 06 20:23:56.813374 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.813353    2563 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-141-123.ec2.internal"
Jul 06 20:23:56.813441 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:23:56.813379    2563 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-141-123.ec2.internal\": node \"ip-10-0-141-123.ec2.internal\" not found"
Jul 06 20:23:56.826156 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.826130    2563 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 06 20:23:56.827524 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.827474    2563 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 06 20:23:56.827524 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.827521    2563 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 06 20:23:56.827690 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.827547    2563 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 06 20:23:56.827690 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:23:56.827592    2563 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 06 20:23:56.829838 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.829811    2563 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 06 20:23:56.835251 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:23:56.835233    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-123.ec2.internal\" not found"
Jul 06 20:23:56.928895 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.928844    2563 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-123.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-123.ec2.internal"]
Jul 06 20:23:56.929045 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.928947    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 06 20:23:56.930038 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.930019    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-123.ec2.internal" event="NodeHasSufficientMemory"
Jul 06 20:23:56.930124 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.930054    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-123.ec2.internal" event="NodeHasNoDiskPressure"
Jul 06 20:23:56.930124 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.930065    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-123.ec2.internal" event="NodeHasSufficientPID"
Jul 06 20:23:56.931541 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.931526    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 06 20:23:56.931679 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.931662    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-123.ec2.internal"
Jul 06 20:23:56.931741 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.931704    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 06 20:23:56.932300 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.932284    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-123.ec2.internal" event="NodeHasSufficientMemory"
Jul 06 20:23:56.932372 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.932283    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-123.ec2.internal" event="NodeHasSufficientMemory"
Jul 06 20:23:56.932372 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.932331    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-123.ec2.internal" event="NodeHasNoDiskPressure"
Jul 06 20:23:56.932372 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.932342    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-123.ec2.internal" event="NodeHasSufficientPID"
Jul 06 20:23:56.932372 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.932309    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-123.ec2.internal" event="NodeHasNoDiskPressure"
Jul 06 20:23:56.932494 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.932392    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-123.ec2.internal" event="NodeHasSufficientPID"
Jul 06 20:23:56.933777 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.933761    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-123.ec2.internal"
Jul 06 20:23:56.933837 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.933791    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 06 20:23:56.934514 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.934498    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-123.ec2.internal" event="NodeHasSufficientMemory"
Jul 06 20:23:56.934605 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.934529    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-123.ec2.internal" event="NodeHasNoDiskPressure"
Jul 06 20:23:56.934605 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.934543    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-123.ec2.internal" event="NodeHasSufficientPID"
Jul 06 20:23:56.935686 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:23:56.935669    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-123.ec2.internal\" not found"
Jul 06 20:23:56.941425 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.941409    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bc8b346ff26bd2da637db45021bfde3d-config\") pod \"kube-apiserver-proxy-ip-10-0-141-123.ec2.internal\" (UID: \"bc8b346ff26bd2da637db45021bfde3d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-123.ec2.internal"
Jul 06 20:23:56.941490 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.941432    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b7c519ecbb7e86b3494895d9cd797fde-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-123.ec2.internal\" (UID: \"b7c519ecbb7e86b3494895d9cd797fde\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-123.ec2.internal"
Jul 06 20:23:56.941490 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:56.941452    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b7c519ecbb7e86b3494895d9cd797fde-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-123.ec2.internal\" (UID: \"b7c519ecbb7e86b3494895d9cd797fde\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-123.ec2.internal"
Jul 06 20:23:56.960538 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:23:56.960515    2563 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-123.ec2.internal\" not found" node="ip-10-0-141-123.ec2.internal"
Jul 06 20:23:56.965304 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:23:56.965286    2563 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-123.ec2.internal\" not found" node="ip-10-0-141-123.ec2.internal"
Jul 06 20:23:57.036189 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:23:57.036149    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-123.ec2.internal\" not found"
Jul 06 20:23:57.041840 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:57.041799    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bc8b346ff26bd2da637db45021bfde3d-config\") pod \"kube-apiserver-proxy-ip-10-0-141-123.ec2.internal\" (UID: \"bc8b346ff26bd2da637db45021bfde3d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-123.ec2.internal"
Jul 06 20:23:57.041971 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:57.041848    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bc8b346ff26bd2da637db45021bfde3d-config\") pod \"kube-apiserver-proxy-ip-10-0-141-123.ec2.internal\" (UID: \"bc8b346ff26bd2da637db45021bfde3d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-123.ec2.internal"
Jul 06 20:23:57.041971 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:57.041861    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b7c519ecbb7e86b3494895d9cd797fde-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-123.ec2.internal\" (UID: \"b7c519ecbb7e86b3494895d9cd797fde\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-123.ec2.internal"
Jul 06 20:23:57.041971 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:57.041910    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b7c519ecbb7e86b3494895d9cd797fde-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-123.ec2.internal\" (UID: \"b7c519ecbb7e86b3494895d9cd797fde\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-123.ec2.internal"
Jul 06 20:23:57.041971 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:57.041906    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b7c519ecbb7e86b3494895d9cd797fde-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-123.ec2.internal\" (UID: \"b7c519ecbb7e86b3494895d9cd797fde\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-123.ec2.internal"
Jul 06 20:23:57.042112 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:57.041970    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b7c519ecbb7e86b3494895d9cd797fde-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-123.ec2.internal\" (UID: \"b7c519ecbb7e86b3494895d9cd797fde\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-123.ec2.internal"
Jul 06 20:23:57.136718 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:23:57.136637    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-123.ec2.internal\" not found"
Jul 06 20:23:57.240572 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:23:57.240515    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-123.ec2.internal\" not found"
Jul 06 20:23:57.264652 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:57.264627    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-123.ec2.internal"
Jul 06 20:23:57.269247 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:57.268888    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-123.ec2.internal"
Jul 06 20:23:57.341316 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:23:57.341272    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-123.ec2.internal\" not found"
Jul 06 20:23:57.441467 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:23:57.441394    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-123.ec2.internal\" not found"
Jul 06 20:23:57.528362 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:57.528332    2563 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 06 20:23:57.542312 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:23:57.542290    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-123.ec2.internal\" not found"
Jul 06 20:23:57.580177 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:57.580159    2563 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 06 20:23:57.580657 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:57.580309    2563 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 06 20:23:57.580657 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:57.580334    2563 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 06 20:23:57.643241 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:23:57.643216    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-123.ec2.internal\" not found"
Jul 06 20:23:57.659695 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:57.659658    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-05 20:18:57 +0000 UTC" deadline="2028-02-08 14:48:18.425276287 +0000 UTC"
Jul 06 20:23:57.659695 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:57.659688    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13962h24m20.765591343s"
Jul 06 20:23:57.743723 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:23:57.743640    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-123.ec2.internal\" not found"
Jul 06 20:23:57.777573 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:57.777546    2563 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 06 20:23:57.844047 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:23:57.844017    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-123.ec2.internal\" not found"
Jul 06 20:23:57.887213 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:57.887191    2563 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 06 20:23:57.895760 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:57.895713    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc8b346ff26bd2da637db45021bfde3d.slice/crio-f3c2d5739c041a947f2cd972d9acae810cf94aaec152a93d782f0642f506ec08 WatchSource:0}: Error finding container f3c2d5739c041a947f2cd972d9acae810cf94aaec152a93d782f0642f506ec08: Status 404 returned error can't find the container with id f3c2d5739c041a947f2cd972d9acae810cf94aaec152a93d782f0642f506ec08
Jul 06 20:23:57.895979 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:23:57.895958    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7c519ecbb7e86b3494895d9cd797fde.slice/crio-2f921f8141c3fe069aa30e7ecbd9b5a43803c407d29716b9fb80f130116af3a0 WatchSource:0}: Error finding container 2f921f8141c3fe069aa30e7ecbd9b5a43803c407d29716b9fb80f130116af3a0: Status 404 returned error can't find the container with id 2f921f8141c3fe069aa30e7ecbd9b5a43803c407d29716b9fb80f130116af3a0
Jul 06 20:23:57.901293 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:57.901270    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 06 20:23:57.941924 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:57.941886    2563 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-123.ec2.internal"
Jul 06 20:23:57.970356 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:57.970326    2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 06 20:23:57.971282 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:57.971265    2563 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-123.ec2.internal"
Jul 06 20:23:57.982837 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:57.982815    2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 06 20:23:58.461076 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.461041    2563 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 06 20:23:58.622024 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.621982    2563 apiserver.go:52] "Watching apiserver"
Jul 06 20:23:58.635756 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.635727    2563 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 06 20:23:58.639597 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.639517    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-gl5px","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ls5cs","openshift-cluster-node-tuning-operator/tuned-7pkrs","openshift-multus/network-metrics-daemon-gvd8v","openshift-network-diagnostics/network-check-target-tc5ls","openshift-network-operator/iptables-alerter-48m7n","openshift-ovn-kubernetes/ovnkube-node-5tk9w","kube-system/global-pull-secret-syncer-nzf7w","kube-system/konnectivity-agent-2cqxl","kube-system/kube-apiserver-proxy-ip-10-0-141-123.ec2.internal","openshift-dns/node-resolver-n9568","openshift-image-registry/node-ca-2hwd9","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-123.ec2.internal","openshift-multus/multus-additional-cni-plugins-b7wcr"]
Jul 06 20:23:58.641633 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.641608    2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 06 20:23:58.642076 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.642053    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-5tk9w"
Jul 06 20:23:58.643537 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.643514    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-n9568"
Jul 06 20:23:58.645237 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.645216    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-7pkrs"
Jul 06 20:23:58.646732 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.646458    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gvd8v"
Jul 06 20:23:58.646732 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:23:58.646541    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gvd8v" podUID="b20534ab-b8c8-4c81-a0f4-0f98a738896e"
Jul 06 20:23:58.648295 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.648171    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 06 20:23:58.648417 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.648397    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tc5ls"
Jul 06 20:23:58.648484 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:23:58.648467    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tc5ls" podUID="d2969391-caa9-4c3d-932c-3727d57b6ce8"
Jul 06 20:23:58.650065 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.649951    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-5cjxm\""
Jul 06 20:23:58.650181 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.650069    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-l88j2\""
Jul 06 20:23:58.650246 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.650186    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-b7wcr"
Jul 06 20:23:58.650582 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.650302    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 06 20:23:58.650582 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.650387    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-zwd7n\""
Jul 06 20:23:58.650582 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.650470    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 06 20:23:58.654211 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.652940    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1816531c-ac4a-4d49-96ac-234daa36ea91-tmp-dir\") pod \"node-resolver-n9568\" (UID: \"1816531c-ac4a-4d49-96ac-234daa36ea91\") " pod="openshift-dns/node-resolver-n9568"
Jul 06 20:23:58.654211 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.652976    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/53799812-6492-4e52-94be-ce6e529cfc88-run-openvswitch\") pod \"ovnkube-node-5tk9w\" (UID: \"53799812-6492-4e52-94be-ce6e529cfc88\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tk9w"
Jul 06 20:23:58.654211 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.653009    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/53799812-6492-4e52-94be-ce6e529cfc88-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5tk9w\" (UID: \"53799812-6492-4e52-94be-ce6e529cfc88\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tk9w"
Jul 06 20:23:58.654211 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.653034    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/53799812-6492-4e52-94be-ce6e529cfc88-ovnkube-config\") pod \"ovnkube-node-5tk9w\" (UID: \"53799812-6492-4e52-94be-ce6e529cfc88\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tk9w"
Jul 06 20:23:58.654211 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.653060    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/53799812-6492-4e52-94be-ce6e529cfc88-ovn-node-metrics-cert\") pod \"ovnkube-node-5tk9w\" (UID: \"53799812-6492-4e52-94be-ce6e529cfc88\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tk9w"
Jul 06 20:23:58.654211 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.653086    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/53799812-6492-4e52-94be-ce6e529cfc88-ovnkube-script-lib\") pod \"ovnkube-node-5tk9w\" (UID: \"53799812-6492-4e52-94be-ce6e529cfc88\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tk9w"
Jul 06 20:23:58.654211 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.653039    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 06 20:23:58.654211 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.653120    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/53799812-6492-4e52-94be-ce6e529cfc88-systemd-units\") pod \"ovnkube-node-5tk9w\" (UID: \"53799812-6492-4e52-94be-ce6e529cfc88\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tk9w"
Jul 06 20:23:58.654211 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.653151    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/53799812-6492-4e52-94be-ce6e529cfc88-var-lib-openvswitch\") pod \"ovnkube-node-5tk9w\" (UID: \"53799812-6492-4e52-94be-ce6e529cfc88\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tk9w"
Jul 06 20:23:58.654211 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.653176    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/53799812-6492-4e52-94be-ce6e529cfc88-run-ovn\") pod \"ovnkube-node-5tk9w\" (UID: \"53799812-6492-4e52-94be-ce6e529cfc88\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tk9w"
Jul 06 20:23:58.654211 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.653213    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/53799812-6492-4e52-94be-ce6e529cfc88-host-run-ovn-kubernetes\") pod \"ovnkube-node-5tk9w\" (UID: \"53799812-6492-4e52-94be-ce6e529cfc88\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tk9w"
Jul 06 20:23:58.654211 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.653238    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/53799812-6492-4e52-94be-ce6e529cfc88-host-cni-bin\") pod \"ovnkube-node-5tk9w\" (UID: \"53799812-6492-4e52-94be-ce6e529cfc88\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tk9w"
Jul 06 20:23:58.654211 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.653259    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1816531c-ac4a-4d49-96ac-234daa36ea91-hosts-file\") pod \"node-resolver-n9568\" (UID: \"1816531c-ac4a-4d49-96ac-234daa36ea91\") " pod="openshift-dns/node-resolver-n9568"
Jul 06 20:23:58.654211 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.653284    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9csg4\" (UniqueName: \"kubernetes.io/projected/1816531c-ac4a-4d49-96ac-234daa36ea91-kube-api-access-9csg4\") pod \"node-resolver-n9568\" (UID: \"1816531c-ac4a-4d49-96ac-234daa36ea91\") " pod="openshift-dns/node-resolver-n9568"
Jul 06 20:23:58.654211 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.653324    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/53799812-6492-4e52-94be-ce6e529cfc88-host-kubelet\") pod \"ovnkube-node-5tk9w\" (UID: \"53799812-6492-4e52-94be-ce6e529cfc88\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tk9w"
Jul 06 20:23:58.654211 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.653352    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/53799812-6492-4e52-94be-ce6e529cfc88-host-slash\") pod \"ovnkube-node-5tk9w\" (UID: \"53799812-6492-4e52-94be-ce6e529cfc88\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tk9w"
Jul 06 20:23:58.654211 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.653375    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/53799812-6492-4e52-94be-ce6e529cfc88-etc-openvswitch\") pod \"ovnkube-node-5tk9w\" (UID: \"53799812-6492-4e52-94be-ce6e529cfc88\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tk9w"
Jul 06 20:23:58.655078 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.653408    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/53799812-6492-4e52-94be-ce6e529cfc88-node-log\") pod \"ovnkube-node-5tk9w\" (UID: \"53799812-6492-4e52-94be-ce6e529cfc88\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tk9w"
Jul 06 20:23:58.655078 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.653432    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/53799812-6492-4e52-94be-ce6e529cfc88-log-socket\") pod \"ovnkube-node-5tk9w\" (UID: \"53799812-6492-4e52-94be-ce6e529cfc88\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tk9w"
Jul 06 20:23:58.655078 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.653457    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/53799812-6492-4e52-94be-ce6e529cfc88-env-overrides\") pod \"ovnkube-node-5tk9w\" (UID: \"53799812-6492-4e52-94be-ce6e529cfc88\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tk9w"
Jul 06 20:23:58.655078 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.653482    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/53799812-6492-4e52-94be-ce6e529cfc88-host-run-netns\") pod \"ovnkube-node-5tk9w\" (UID: \"53799812-6492-4e52-94be-ce6e529cfc88\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tk9w"
Jul 06 20:23:58.655078 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.653499    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-2cqxl"
Jul 06 20:23:58.655078 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.653506    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/53799812-6492-4e52-94be-ce6e529cfc88-run-systemd\") pod \"ovnkube-node-5tk9w\" (UID: \"53799812-6492-4e52-94be-ce6e529cfc88\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tk9w"
Jul 06 20:23:58.655078 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.653530    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/53799812-6492-4e52-94be-ce6e529cfc88-host-cni-netd\") pod \"ovnkube-node-5tk9w\" (UID: \"53799812-6492-4e52-94be-ce6e529cfc88\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tk9w"
Jul 06 20:23:58.655078 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.653555    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2m6bd\" (UniqueName: \"kubernetes.io/projected/53799812-6492-4e52-94be-ce6e529cfc88-kube-api-access-2m6bd\") pod \"ovnkube-node-5tk9w\" (UID: \"53799812-6492-4e52-94be-ce6e529cfc88\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tk9w"
Jul 06 20:23:58.655078 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.654264    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-2hwd9"
Jul 06 20:23:58.656079 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.656061    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-gl5px"
Jul 06 20:23:58.657830 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.657667    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ls5cs"
Jul 06 20:23:58.660245 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.659917    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nzf7w"
Jul 06 20:23:58.660245 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:23:58.659979    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nzf7w" podUID="e42d3c32-18ba-48dc-afb5-90ef7f0e2752"
Jul 06 20:23:58.662005 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.661984    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-48m7n"
Jul 06 20:23:58.670339 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.670285    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 06 20:23:58.670579 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.670564    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 06 20:23:58.670657 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.670607    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 06 20:23:58.670833 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.670780    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 06 20:23:58.670914 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.670882    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 06 20:23:58.676578 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.676558    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 06 20:23:58.678981 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.678961    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-gfv9b\""
Jul 06 20:23:58.679286 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.679266    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 06 20:23:58.698291 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.698055    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 06 20:23:58.698291 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.698055    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 06 20:23:58.698456 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.698393    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 06 20:23:58.698568 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.698541    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 06 20:23:58.699569 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.699548    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 06 20:23:58.699924 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.699903    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 06 20:23:58.700066 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.699926    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 06 20:23:58.700411 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.700392    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-r85lv\""
Jul 06 20:23:58.700658 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.700638    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 06 20:23:58.700724 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.700676    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 06 20:23:58.700818 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.700793    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 06 20:23:58.701072 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.701051    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 06 20:23:58.701676 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.701654    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 06 20:23:58.701754 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.701685    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 06 20:23:58.701977 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.701957    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 06 20:23:58.702092 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.702034    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 06 20:23:58.703215 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.703186    2563 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 06 20:23:58.703339 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.703323    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-mnw8b\""
Jul 06 20:23:58.703415 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.703355    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-dvklp\""
Jul 06 20:23:58.703668 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.703639    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-hhwzt\""
Jul 06 20:23:58.703668 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.703664    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-pbdcq\""
Jul 06 20:23:58.703990 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.703935    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 06 20:23:58.713007 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.712955    2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-bm8wx"
Jul 06 20:23:58.722043 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.722009    2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-bm8wx"
Jul 06 20:23:58.742767 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.742739    2563 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 06 20:23:58.754491 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.754466    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/048dd56c-9989-44f2-a1d5-c3c50db6938c-cni-binary-copy\") pod \"multus-additional-cni-plugins-b7wcr\" (UID: \"048dd56c-9989-44f2-a1d5-c3c50db6938c\") " pod="openshift-multus/multus-additional-cni-plugins-b7wcr"
Jul 06 20:23:58.754634 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.754510    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/53799812-6492-4e52-94be-ce6e529cfc88-host-kubelet\") pod \"ovnkube-node-5tk9w\" (UID: \"53799812-6492-4e52-94be-ce6e529cfc88\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tk9w"
Jul 06 20:23:58.754634 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.754573    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/53799812-6492-4e52-94be-ce6e529cfc88-env-overrides\") pod \"ovnkube-node-5tk9w\" (UID: \"53799812-6492-4e52-94be-ce6e529cfc88\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tk9w"
Jul 06 20:23:58.754634 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.754602    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kqkd\" (UniqueName: \"kubernetes.io/projected/66288c77-335a-4c17-9ba7-bee3609820cb-kube-api-access-4kqkd\") pod \"aws-ebs-csi-driver-node-ls5cs\" (UID: \"66288c77-335a-4c17-9ba7-bee3609820cb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ls5cs"
Jul 06 20:23:58.754634 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.754607    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/53799812-6492-4e52-94be-ce6e529cfc88-host-kubelet\") pod \"ovnkube-node-5tk9w\" (UID: \"53799812-6492-4e52-94be-ce6e529cfc88\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tk9w"
Jul 06 20:23:58.754877 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.754640    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f48f979d-1c07-47ef-8144-488cad085d6e-etc-sysctl-d\") pod \"tuned-7pkrs\" (UID: \"f48f979d-1c07-47ef-8144-488cad085d6e\") " pod="openshift-cluster-node-tuning-operator/tuned-7pkrs"
Jul 06 20:23:58.754877 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.754665    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f48f979d-1c07-47ef-8144-488cad085d6e-run\") pod \"tuned-7pkrs\" (UID: \"f48f979d-1c07-47ef-8144-488cad085d6e\") " pod="openshift-cluster-node-tuning-operator/tuned-7pkrs"
Jul 06 20:23:58.754877 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.754691    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5xth\" (UniqueName: \"kubernetes.io/projected/b20534ab-b8c8-4c81-a0f4-0f98a738896e-kube-api-access-x5xth\") pod \"network-metrics-daemon-gvd8v\" (UID: \"b20534ab-b8c8-4c81-a0f4-0f98a738896e\") " pod="openshift-multus/network-metrics-daemon-gvd8v"
Jul 06 20:23:58.754877 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.754716    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/64a3d8f1-f091-4ba3-9f90-1b77cfea3c71-multus-daemon-config\") pod \"multus-gl5px\" (UID: \"64a3d8f1-f091-4ba3-9f90-1b77cfea3c71\") " pod="openshift-multus/multus-gl5px"
Jul 06 20:23:58.754877 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.754739    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/64a3d8f1-f091-4ba3-9f90-1b77cfea3c71-host-run-multus-certs\") pod \"multus-gl5px\" (UID: \"64a3d8f1-f091-4ba3-9f90-1b77cfea3c71\") " pod="openshift-multus/multus-gl5px"
Jul 06 20:23:58.754877 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.754780    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7f8a1214-4b3e-4de6-96fe-8f9be6cca7f7-iptables-alerter-script\") pod \"iptables-alerter-48m7n\" (UID: \"7f8a1214-4b3e-4de6-96fe-8f9be6cca7f7\") " pod="openshift-network-operator/iptables-alerter-48m7n"
Jul 06 20:23:58.754877 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.754803    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f48f979d-1c07-47ef-8144-488cad085d6e-etc-systemd\") pod \"tuned-7pkrs\" (UID: \"f48f979d-1c07-47ef-8144-488cad085d6e\") " pod="openshift-cluster-node-tuning-operator/tuned-7pkrs"
Jul 06 20:23:58.754877 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.754861    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9wr4\" (UniqueName: \"kubernetes.io/projected/d2969391-caa9-4c3d-932c-3727d57b6ce8-kube-api-access-m9wr4\") pod \"network-check-target-tc5ls\" (UID: \"d2969391-caa9-4c3d-932c-3727d57b6ce8\") " pod="openshift-network-diagnostics/network-check-target-tc5ls"
Jul 06 20:23:58.755268 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.754902    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwgtw\" (UniqueName: \"kubernetes.io/projected/048dd56c-9989-44f2-a1d5-c3c50db6938c-kube-api-access-gwgtw\") pod \"multus-additional-cni-plugins-b7wcr\" (UID: \"048dd56c-9989-44f2-a1d5-c3c50db6938c\") " pod="openshift-multus/multus-additional-cni-plugins-b7wcr"
Jul 06 20:23:58.755268 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.754929    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/53799812-6492-4e52-94be-ce6e529cfc88-run-openvswitch\") pod \"ovnkube-node-5tk9w\" (UID: \"53799812-6492-4e52-94be-ce6e529cfc88\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tk9w"
Jul 06 20:23:58.755268 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.754948    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/53799812-6492-4e52-94be-ce6e529cfc88-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5tk9w\" (UID: \"53799812-6492-4e52-94be-ce6e529cfc88\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tk9w"
Jul 06 20:23:58.755268 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.754966    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/408f722a-5e7e-4bad-a86e-d461b5fed747-konnectivity-ca\") pod \"konnectivity-agent-2cqxl\" (UID: \"408f722a-5e7e-4bad-a86e-d461b5fed747\") " pod="kube-system/konnectivity-agent-2cqxl"
Jul 06 20:23:58.755268 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.754999    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f48f979d-1c07-47ef-8144-488cad085d6e-etc-sysconfig\") pod \"tuned-7pkrs\" (UID: \"f48f979d-1c07-47ef-8144-488cad085d6e\") " pod="openshift-cluster-node-tuning-operator/tuned-7pkrs"
Jul 06 20:23:58.755268 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.755026    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/048dd56c-9989-44f2-a1d5-c3c50db6938c-os-release\") pod \"multus-additional-cni-plugins-b7wcr\" (UID: \"048dd56c-9989-44f2-a1d5-c3c50db6938c\") " pod="openshift-multus/multus-additional-cni-plugins-b7wcr"
Jul 06 20:23:58.755268 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.755053    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/048dd56c-9989-44f2-a1d5-c3c50db6938c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-b7wcr\" (UID: \"048dd56c-9989-44f2-a1d5-c3c50db6938c\") " pod="openshift-multus/multus-additional-cni-plugins-b7wcr"
Jul 06 20:23:58.755268 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.755060    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/53799812-6492-4e52-94be-ce6e529cfc88-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5tk9w\" (UID: \"53799812-6492-4e52-94be-ce6e529cfc88\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tk9w"
Jul 06 20:23:58.755268 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.755028    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/53799812-6492-4e52-94be-ce6e529cfc88-run-openvswitch\") pod \"ovnkube-node-5tk9w\" (UID: \"53799812-6492-4e52-94be-ce6e529cfc88\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tk9w"
Jul 06 20:23:58.755268 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.755101    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/408f722a-5e7e-4bad-a86e-d461b5fed747-agent-certs\") pod \"konnectivity-agent-2cqxl\" (UID: \"408f722a-5e7e-4bad-a86e-d461b5fed747\") " pod="kube-system/konnectivity-agent-2cqxl"
Jul 06 20:23:58.755268 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.755148    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kx5c4\" (UniqueName: \"kubernetes.io/projected/ed41c82c-bbcc-4385-8f89-f6ff340c2e36-kube-api-access-kx5c4\") pod \"node-ca-2hwd9\" (UID: \"ed41c82c-bbcc-4385-8f89-f6ff340c2e36\") " pod="openshift-image-registry/node-ca-2hwd9"
Jul 06 20:23:58.755268 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.755177    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/048dd56c-9989-44f2-a1d5-c3c50db6938c-system-cni-dir\") pod \"multus-additional-cni-plugins-b7wcr\" (UID: \"048dd56c-9989-44f2-a1d5-c3c50db6938c\") " pod="openshift-multus/multus-additional-cni-plugins-b7wcr"
Jul 06 20:23:58.755268 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.755191    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/53799812-6492-4e52-94be-ce6e529cfc88-env-overrides\") pod \"ovnkube-node-5tk9w\" (UID: \"53799812-6492-4e52-94be-ce6e529cfc88\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tk9w"
Jul 06 20:23:58.755268 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.755211    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/53799812-6492-4e52-94be-ce6e529cfc88-etc-openvswitch\") pod \"ovnkube-node-5tk9w\" (UID: \"53799812-6492-4e52-94be-ce6e529cfc88\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tk9w"
Jul 06 20:23:58.755268 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.755239    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/53799812-6492-4e52-94be-ce6e529cfc88-node-log\") pod \"ovnkube-node-5tk9w\" (UID: \"53799812-6492-4e52-94be-ce6e529cfc88\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tk9w"
Jul 06 20:23:58.756035 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.755290    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/53799812-6492-4e52-94be-ce6e529cfc88-node-log\") pod \"ovnkube-node-5tk9w\" (UID: \"53799812-6492-4e52-94be-ce6e529cfc88\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tk9w"
Jul 06 20:23:58.756035 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.755286    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/53799812-6492-4e52-94be-ce6e529cfc88-log-socket\") pod \"ovnkube-node-5tk9w\" (UID: \"53799812-6492-4e52-94be-ce6e529cfc88\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tk9w"
Jul 06 20:23:58.756035 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.755324    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/64a3d8f1-f091-4ba3-9f90-1b77cfea3c71-os-release\") pod \"multus-gl5px\" (UID: \"64a3d8f1-f091-4ba3-9f90-1b77cfea3c71\") " pod="openshift-multus/multus-gl5px"
Jul 06 20:23:58.756035 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.755332    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/53799812-6492-4e52-94be-ce6e529cfc88-log-socket\") pod \"ovnkube-node-5tk9w\" (UID: \"53799812-6492-4e52-94be-ce6e529cfc88\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tk9w"
Jul 06 20:23:58.756035 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.755351    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sf5ls\" (UniqueName: \"kubernetes.io/projected/64a3d8f1-f091-4ba3-9f90-1b77cfea3c71-kube-api-access-sf5ls\") pod \"multus-gl5px\" (UID: \"64a3d8f1-f091-4ba3-9f90-1b77cfea3c71\") " pod="openshift-multus/multus-gl5px"
Jul 06 20:23:58.756035 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.755364    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/53799812-6492-4e52-94be-ce6e529cfc88-etc-openvswitch\") pod \"ovnkube-node-5tk9w\" (UID: \"53799812-6492-4e52-94be-ce6e529cfc88\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tk9w"
Jul 06 20:23:58.756035 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.755386    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e42d3c32-18ba-48dc-afb5-90ef7f0e2752-kubelet-config\") pod \"global-pull-secret-syncer-nzf7w\" (UID: \"e42d3c32-18ba-48dc-afb5-90ef7f0e2752\") " pod="kube-system/global-pull-secret-syncer-nzf7w"
Jul 06 20:23:58.756035 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.755444    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f48f979d-1c07-47ef-8144-488cad085d6e-tmp\") pod \"tuned-7pkrs\" (UID: \"f48f979d-1c07-47ef-8144-488cad085d6e\") " pod="openshift-cluster-node-tuning-operator/tuned-7pkrs"
Jul 06 20:23:58.756035 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.755473    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/53799812-6492-4e52-94be-ce6e529cfc88-host-run-netns\") pod \"ovnkube-node-5tk9w\" (UID: \"53799812-6492-4e52-94be-ce6e529cfc88\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tk9w"
Jul 06 20:23:58.756035 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.755500    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1816531c-ac4a-4d49-96ac-234daa36ea91-tmp-dir\") pod \"node-resolver-n9568\" (UID: \"1816531c-ac4a-4d49-96ac-234daa36ea91\") " pod="openshift-dns/node-resolver-n9568"
Jul 06 20:23:58.756035 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.755525    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/64a3d8f1-f091-4ba3-9f90-1b77cfea3c71-multus-cni-dir\") pod \"multus-gl5px\" (UID: \"64a3d8f1-f091-4ba3-9f90-1b77cfea3c71\") " pod="openshift-multus/multus-gl5px"
Jul 06 20:23:58.756035 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.755553    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/64a3d8f1-f091-4ba3-9f90-1b77cfea3c71-host-var-lib-cni-multus\") pod \"multus-gl5px\" (UID: \"64a3d8f1-f091-4ba3-9f90-1b77cfea3c71\") " pod="openshift-multus/multus-gl5px"
Jul 06 20:23:58.756035 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.755575    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f48f979d-1c07-47ef-8144-488cad085d6e-host\") pod \"tuned-7pkrs\" (UID: \"f48f979d-1c07-47ef-8144-488cad085d6e\") " pod="openshift-cluster-node-tuning-operator/tuned-7pkrs"
Jul 06 20:23:58.756035 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.755664    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/048dd56c-9989-44f2-a1d5-c3c50db6938c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-b7wcr\" (UID: \"048dd56c-9989-44f2-a1d5-c3c50db6938c\") " pod="openshift-multus/multus-additional-cni-plugins-b7wcr"
Jul 06 20:23:58.756035 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.755694    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/53799812-6492-4e52-94be-ce6e529cfc88-ovn-node-metrics-cert\") pod \"ovnkube-node-5tk9w\" (UID: \"53799812-6492-4e52-94be-ce6e529cfc88\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tk9w"
Jul 06 20:23:58.756035 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.755718    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/66288c77-335a-4c17-9ba7-bee3609820cb-sys-fs\") pod \"aws-ebs-csi-driver-node-ls5cs\" (UID: \"66288c77-335a-4c17-9ba7-bee3609820cb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ls5cs"
Jul 06 20:23:58.756035 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.755743    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/64a3d8f1-f091-4ba3-9f90-1b77cfea3c71-cni-binary-copy\") pod \"multus-gl5px\" (UID: \"64a3d8f1-f091-4ba3-9f90-1b77cfea3c71\") " pod="openshift-multus/multus-gl5px"
Jul 06 20:23:58.757003 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.755783    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e42d3c32-18ba-48dc-afb5-90ef7f0e2752-original-pull-secret\") pod \"global-pull-secret-syncer-nzf7w\" (UID: \"e42d3c32-18ba-48dc-afb5-90ef7f0e2752\") " pod="kube-system/global-pull-secret-syncer-nzf7w"
Jul 06 20:23:58.757003 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.755835    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/53799812-6492-4e52-94be-ce6e529cfc88-systemd-units\") pod \"ovnkube-node-5tk9w\" (UID: \"53799812-6492-4e52-94be-ce6e529cfc88\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tk9w"
Jul 06 20:23:58.757003 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.755861    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/53799812-6492-4e52-94be-ce6e529cfc88-run-ovn\") pod \"ovnkube-node-5tk9w\" (UID: \"53799812-6492-4e52-94be-ce6e529cfc88\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tk9w"
Jul 06 20:23:58.757003 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.755863    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1816531c-ac4a-4d49-96ac-234daa36ea91-tmp-dir\") pod \"node-resolver-n9568\" (UID: \"1816531c-ac4a-4d49-96ac-234daa36ea91\") " pod="openshift-dns/node-resolver-n9568"
Jul 06 20:23:58.757003 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.755884    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/53799812-6492-4e52-94be-ce6e529cfc88-host-cni-bin\") pod \"ovnkube-node-5tk9w\" (UID: \"53799812-6492-4e52-94be-ce6e529cfc88\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tk9w"
Jul 06 20:23:58.757003 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.755913    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1816531c-ac4a-4d49-96ac-234daa36ea91-hosts-file\") pod \"node-resolver-n9568\" (UID: \"1816531c-ac4a-4d49-96ac-234daa36ea91\") " pod="openshift-dns/node-resolver-n9568"
Jul 06 20:23:58.757003 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.755947    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ed41c82c-bbcc-4385-8f89-f6ff340c2e36-serviceca\") pod \"node-ca-2hwd9\" (UID: \"ed41c82c-bbcc-4385-8f89-f6ff340c2e36\") " pod="openshift-image-registry/node-ca-2hwd9"
Jul 06 20:23:58.757003 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.755975    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/64a3d8f1-f091-4ba3-9f90-1b77cfea3c71-multus-socket-dir-parent\") pod \"multus-gl5px\" (UID: \"64a3d8f1-f091-4ba3-9f90-1b77cfea3c71\") " pod="openshift-multus/multus-gl5px"
Jul 06 20:23:58.757003 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.756001    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7f8a1214-4b3e-4de6-96fe-8f9be6cca7f7-host-slash\") pod \"iptables-alerter-48m7n\" (UID: \"7f8a1214-4b3e-4de6-96fe-8f9be6cca7f7\") " pod="openshift-network-operator/iptables-alerter-48m7n"
Jul 06 20:23:58.757003 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.756036    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f48f979d-1c07-47ef-8144-488cad085d6e-etc-modprobe-d\") pod \"tuned-7pkrs\" (UID: \"f48f979d-1c07-47ef-8144-488cad085d6e\") " pod="openshift-cluster-node-tuning-operator/tuned-7pkrs"
Jul 06 20:23:58.757003 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.756062    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ed41c82c-bbcc-4385-8f89-f6ff340c2e36-host\") pod \"node-ca-2hwd9\" (UID: \"ed41c82c-bbcc-4385-8f89-f6ff340c2e36\") " pod="openshift-image-registry/node-ca-2hwd9"
Jul 06 20:23:58.757003 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.756088    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/64a3d8f1-f091-4ba3-9f90-1b77cfea3c71-host-run-netns\") pod \"multus-gl5px\" (UID: \"64a3d8f1-f091-4ba3-9f90-1b77cfea3c71\") " pod="openshift-multus/multus-gl5px"
Jul 06 20:23:58.757003 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.756101    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/53799812-6492-4e52-94be-ce6e529cfc88-run-ovn\") pod \"ovnkube-node-5tk9w\" (UID: \"53799812-6492-4e52-94be-ce6e529cfc88\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tk9w"
Jul 06 20:23:58.757003 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.756128    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lm8dh\" (UniqueName: \"kubernetes.io/projected/7f8a1214-4b3e-4de6-96fe-8f9be6cca7f7-kube-api-access-lm8dh\") pod \"iptables-alerter-48m7n\" (UID: \"7f8a1214-4b3e-4de6-96fe-8f9be6cca7f7\") " pod="openshift-network-operator/iptables-alerter-48m7n"
Jul 06 20:23:58.757003 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.756173    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/53799812-6492-4e52-94be-ce6e529cfc88-systemd-units\") pod \"ovnkube-node-5tk9w\" (UID: \"53799812-6492-4e52-94be-ce6e529cfc88\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tk9w"
Jul 06 20:23:58.757003 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.756191    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/53799812-6492-4e52-94be-ce6e529cfc88-host-cni-bin\") pod \"ovnkube-node-5tk9w\" (UID: \"53799812-6492-4e52-94be-ce6e529cfc88\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tk9w"
Jul 06 20:23:58.757003 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.755524    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/53799812-6492-4e52-94be-ce6e529cfc88-host-run-netns\") pod \"ovnkube-node-5tk9w\" (UID: \"53799812-6492-4e52-94be-ce6e529cfc88\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tk9w"
Jul 06 20:23:58.757874 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.756242    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1816531c-ac4a-4d49-96ac-234daa36ea91-hosts-file\") pod \"node-resolver-n9568\" (UID: \"1816531c-ac4a-4d49-96ac-234daa36ea91\") " pod="openshift-dns/node-resolver-n9568"
Jul 06 20:23:58.757874 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.756248    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f48f979d-1c07-47ef-8144-488cad085d6e-etc-kubernetes\") pod \"tuned-7pkrs\" (UID: \"f48f979d-1c07-47ef-8144-488cad085d6e\") " pod="openshift-cluster-node-tuning-operator/tuned-7pkrs"
Jul 06 20:23:58.757874 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.756251    2563 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 06 20:23:58.757874 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.756292    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tk44x\" (UniqueName: \"kubernetes.io/projected/f48f979d-1c07-47ef-8144-488cad085d6e-kube-api-access-tk44x\") pod \"tuned-7pkrs\" (UID: \"f48f979d-1c07-47ef-8144-488cad085d6e\") " pod="openshift-cluster-node-tuning-operator/tuned-7pkrs"
Jul 06 20:23:58.757874 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.756360    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/048dd56c-9989-44f2-a1d5-c3c50db6938c-cnibin\") pod \"multus-additional-cni-plugins-b7wcr\" (UID: \"048dd56c-9989-44f2-a1d5-c3c50db6938c\") " pod="openshift-multus/multus-additional-cni-plugins-b7wcr"
Jul 06 20:23:58.757874 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.756399    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b20534ab-b8c8-4c81-a0f4-0f98a738896e-metrics-certs\") pod \"network-metrics-daemon-gvd8v\" (UID: \"b20534ab-b8c8-4c81-a0f4-0f98a738896e\") " pod="openshift-multus/network-metrics-daemon-gvd8v"
Jul 06 20:23:58.757874 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.756432    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/53799812-6492-4e52-94be-ce6e529cfc88-ovnkube-config\") pod \"ovnkube-node-5tk9w\" (UID: \"53799812-6492-4e52-94be-ce6e529cfc88\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tk9w"
Jul 06 20:23:58.757874 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.756460    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/66288c77-335a-4c17-9ba7-bee3609820cb-registration-dir\") pod \"aws-ebs-csi-driver-node-ls5cs\" (UID: \"66288c77-335a-4c17-9ba7-bee3609820cb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ls5cs"
Jul 06 20:23:58.757874 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.756508    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/64a3d8f1-f091-4ba3-9f90-1b77cfea3c71-system-cni-dir\") pod \"multus-gl5px\" (UID: \"64a3d8f1-f091-4ba3-9f90-1b77cfea3c71\") " pod="openshift-multus/multus-gl5px"
Jul 06 20:23:58.757874 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.756545    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/64a3d8f1-f091-4ba3-9f90-1b77cfea3c71-cnibin\") pod \"multus-gl5px\" (UID: \"64a3d8f1-f091-4ba3-9f90-1b77cfea3c71\") " pod="openshift-multus/multus-gl5px"
Jul 06 20:23:58.757874 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.756659    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/64a3d8f1-f091-4ba3-9f90-1b77cfea3c71-host-var-lib-cni-bin\") pod \"multus-gl5px\" (UID: \"64a3d8f1-f091-4ba3-9f90-1b77cfea3c71\") " pod="openshift-multus/multus-gl5px"
Jul 06 20:23:58.757874 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.756687    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/64a3d8f1-f091-4ba3-9f90-1b77cfea3c71-etc-kubernetes\") pod \"multus-gl5px\" (UID: \"64a3d8f1-f091-4ba3-9f90-1b77cfea3c71\") " pod="openshift-multus/multus-gl5px"
Jul 06 20:23:58.757874 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.756712    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e42d3c32-18ba-48dc-afb5-90ef7f0e2752-dbus\") pod \"global-pull-secret-syncer-nzf7w\" (UID: \"e42d3c32-18ba-48dc-afb5-90ef7f0e2752\") " pod="kube-system/global-pull-secret-syncer-nzf7w"
Jul 06 20:23:58.757874 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.756770    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f48f979d-1c07-47ef-8144-488cad085d6e-etc-tuned\") pod \"tuned-7pkrs\" (UID: \"f48f979d-1c07-47ef-8144-488cad085d6e\") " pod="openshift-cluster-node-tuning-operator/tuned-7pkrs"
Jul 06 20:23:58.757874 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.756799    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9csg4\" (UniqueName: \"kubernetes.io/projected/1816531c-ac4a-4d49-96ac-234daa36ea91-kube-api-access-9csg4\") pod \"node-resolver-n9568\" (UID: \"1816531c-ac4a-4d49-96ac-234daa36ea91\") " pod="openshift-dns/node-resolver-n9568"
Jul 06 20:23:58.757874 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.756832    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/66288c77-335a-4c17-9ba7-bee3609820cb-socket-dir\") pod \"aws-ebs-csi-driver-node-ls5cs\" (UID: \"66288c77-335a-4c17-9ba7-bee3609820cb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ls5cs"
Jul 06 20:23:58.757874 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.756856    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/64a3d8f1-f091-4ba3-9f90-1b77cfea3c71-host-var-lib-kubelet\") pod \"multus-gl5px\" (UID: \"64a3d8f1-f091-4ba3-9f90-1b77cfea3c71\") " pod="openshift-multus/multus-gl5px"
Jul 06 20:23:58.758736 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.756881    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f48f979d-1c07-47ef-8144-488cad085d6e-var-lib-kubelet\") pod \"tuned-7pkrs\" (UID: \"f48f979d-1c07-47ef-8144-488cad085d6e\") " pod="openshift-cluster-node-tuning-operator/tuned-7pkrs"
Jul 06 20:23:58.758736 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.756906    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/53799812-6492-4e52-94be-ce6e529cfc88-host-slash\") pod \"ovnkube-node-5tk9w\" (UID: \"53799812-6492-4e52-94be-ce6e529cfc88\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tk9w"
Jul 06 20:23:58.758736 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.756983    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2m6bd\" (UniqueName: \"kubernetes.io/projected/53799812-6492-4e52-94be-ce6e529cfc88-kube-api-access-2m6bd\") pod \"ovnkube-node-5tk9w\" (UID: \"53799812-6492-4e52-94be-ce6e529cfc88\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tk9w"
Jul 06 20:23:58.758736 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.757012    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/66288c77-335a-4c17-9ba7-bee3609820cb-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ls5cs\" (UID: \"66288c77-335a-4c17-9ba7-bee3609820cb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ls5cs"
Jul 06 20:23:58.758736 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.757038    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/53799812-6492-4e52-94be-ce6e529cfc88-run-systemd\") pod \"ovnkube-node-5tk9w\" (UID: \"53799812-6492-4e52-94be-ce6e529cfc88\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tk9w"
Jul 06 20:23:58.758736 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.757055    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/53799812-6492-4e52-94be-ce6e529cfc88-ovnkube-config\") pod \"ovnkube-node-5tk9w\" (UID: \"53799812-6492-4e52-94be-ce6e529cfc88\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tk9w"
Jul 06 20:23:58.758736 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.757062    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/53799812-6492-4e52-94be-ce6e529cfc88-host-cni-netd\") pod \"ovnkube-node-5tk9w\" (UID: \"53799812-6492-4e52-94be-ce6e529cfc88\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tk9w"
Jul 06 20:23:58.758736 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.757124    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/66288c77-335a-4c17-9ba7-bee3609820cb-device-dir\") pod \"aws-ebs-csi-driver-node-ls5cs\" (UID: \"66288c77-335a-4c17-9ba7-bee3609820cb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ls5cs"
Jul 06 20:23:58.758736 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.757152    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/64a3d8f1-f091-4ba3-9f90-1b77cfea3c71-host-run-k8s-cni-cncf-io\") pod \"multus-gl5px\" (UID: \"64a3d8f1-f091-4ba3-9f90-1b77cfea3c71\") " pod="openshift-multus/multus-gl5px"
Jul 06 20:23:58.758736 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.757179    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f48f979d-1c07-47ef-8144-488cad085d6e-sys\") pod \"tuned-7pkrs\" (UID: \"f48f979d-1c07-47ef-8144-488cad085d6e\") " pod="openshift-cluster-node-tuning-operator/tuned-7pkrs"
Jul 06 20:23:58.758736 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.757216    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/048dd56c-9989-44f2-a1d5-c3c50db6938c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-b7wcr\" (UID: \"048dd56c-9989-44f2-a1d5-c3c50db6938c\") " pod="openshift-multus/multus-additional-cni-plugins-b7wcr"
Jul 06 20:23:58.758736 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.757259    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/53799812-6492-4e52-94be-ce6e529cfc88-host-slash\") pod \"ovnkube-node-5tk9w\" (UID: \"53799812-6492-4e52-94be-ce6e529cfc88\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tk9w"
Jul 06 20:23:58.758736 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.757304    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/53799812-6492-4e52-94be-ce6e529cfc88-ovnkube-script-lib\") pod \"ovnkube-node-5tk9w\" (UID: \"53799812-6492-4e52-94be-ce6e529cfc88\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tk9w"
Jul 06 20:23:58.758736 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.757329    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/64a3d8f1-f091-4ba3-9f90-1b77cfea3c71-hostroot\") pod \"multus-gl5px\" (UID: \"64a3d8f1-f091-4ba3-9f90-1b77cfea3c71\") " pod="openshift-multus/multus-gl5px"
Jul 06 20:23:58.758736 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.757355    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/64a3d8f1-f091-4ba3-9f90-1b77cfea3c71-multus-conf-dir\") pod \"multus-gl5px\" (UID: \"64a3d8f1-f091-4ba3-9f90-1b77cfea3c71\") " pod="openshift-multus/multus-gl5px"
Jul 06 20:23:58.758736 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.757381    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f48f979d-1c07-47ef-8144-488cad085d6e-lib-modules\") pod \"tuned-7pkrs\" (UID: \"f48f979d-1c07-47ef-8144-488cad085d6e\") " pod="openshift-cluster-node-tuning-operator/tuned-7pkrs"
Jul 06 20:23:58.758736 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.757385    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/53799812-6492-4e52-94be-ce6e529cfc88-host-cni-netd\") pod \"ovnkube-node-5tk9w\" (UID: \"53799812-6492-4e52-94be-ce6e529cfc88\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tk9w"
Jul 06 20:23:58.759534 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.757433    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/53799812-6492-4e52-94be-ce6e529cfc88-run-systemd\") pod \"ovnkube-node-5tk9w\" (UID: \"53799812-6492-4e52-94be-ce6e529cfc88\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tk9w"
Jul 06 20:23:58.759534 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.757529    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/53799812-6492-4e52-94be-ce6e529cfc88-var-lib-openvswitch\") pod \"ovnkube-node-5tk9w\" (UID: \"53799812-6492-4e52-94be-ce6e529cfc88\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tk9w"
Jul 06 20:23:58.759534 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.757591    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/53799812-6492-4e52-94be-ce6e529cfc88-host-run-ovn-kubernetes\") pod \"ovnkube-node-5tk9w\" (UID: \"53799812-6492-4e52-94be-ce6e529cfc88\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tk9w"
Jul 06 20:23:58.759534 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.757631    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/66288c77-335a-4c17-9ba7-bee3609820cb-etc-selinux\") pod \"aws-ebs-csi-driver-node-ls5cs\" (UID: \"66288c77-335a-4c17-9ba7-bee3609820cb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ls5cs"
Jul 06 20:23:58.759534 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.757658    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/53799812-6492-4e52-94be-ce6e529cfc88-host-run-ovn-kubernetes\") pod \"ovnkube-node-5tk9w\" (UID: \"53799812-6492-4e52-94be-ce6e529cfc88\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tk9w"
Jul 06 20:23:58.759534 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.757659    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/53799812-6492-4e52-94be-ce6e529cfc88-var-lib-openvswitch\") pod \"ovnkube-node-5tk9w\" (UID: \"53799812-6492-4e52-94be-ce6e529cfc88\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tk9w"
Jul 06 20:23:58.759534 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.757690    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f48f979d-1c07-47ef-8144-488cad085d6e-etc-sysctl-conf\") pod \"tuned-7pkrs\" (UID: \"f48f979d-1c07-47ef-8144-488cad085d6e\") " pod="openshift-cluster-node-tuning-operator/tuned-7pkrs"
Jul 06 20:23:58.759534 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.758037    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/53799812-6492-4e52-94be-ce6e529cfc88-ovnkube-script-lib\") pod \"ovnkube-node-5tk9w\" (UID: \"53799812-6492-4e52-94be-ce6e529cfc88\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tk9w"
Jul 06 20:23:58.760290 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.760270    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/53799812-6492-4e52-94be-ce6e529cfc88-ovn-node-metrics-cert\") pod \"ovnkube-node-5tk9w\" (UID: \"53799812-6492-4e52-94be-ce6e529cfc88\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tk9w"
Jul 06 20:23:58.769872 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.769813    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9csg4\" (UniqueName: \"kubernetes.io/projected/1816531c-ac4a-4d49-96ac-234daa36ea91-kube-api-access-9csg4\") pod \"node-resolver-n9568\" (UID: \"1816531c-ac4a-4d49-96ac-234daa36ea91\") " pod="openshift-dns/node-resolver-n9568"
Jul 06 20:23:58.770247 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.770226    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2m6bd\" (UniqueName: \"kubernetes.io/projected/53799812-6492-4e52-94be-ce6e529cfc88-kube-api-access-2m6bd\") pod \"ovnkube-node-5tk9w\" (UID: \"53799812-6492-4e52-94be-ce6e529cfc88\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tk9w"
Jul 06 20:23:58.832795 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.832718    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-123.ec2.internal" event={"ID":"bc8b346ff26bd2da637db45021bfde3d","Type":"ContainerStarted","Data":"f3c2d5739c041a947f2cd972d9acae810cf94aaec152a93d782f0642f506ec08"}
Jul 06 20:23:58.834162 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.834130    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-123.ec2.internal" event={"ID":"b7c519ecbb7e86b3494895d9cd797fde","Type":"ContainerStarted","Data":"2f921f8141c3fe069aa30e7ecbd9b5a43803c407d29716b9fb80f130116af3a0"}
Jul 06 20:23:58.858755 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.858719    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/048dd56c-9989-44f2-a1d5-c3c50db6938c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-b7wcr\" (UID: \"048dd56c-9989-44f2-a1d5-c3c50db6938c\") " pod="openshift-multus/multus-additional-cni-plugins-b7wcr"
Jul 06 20:23:58.858907 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.858799    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/408f722a-5e7e-4bad-a86e-d461b5fed747-agent-certs\") pod \"konnectivity-agent-2cqxl\" (UID: \"408f722a-5e7e-4bad-a86e-d461b5fed747\") " pod="kube-system/konnectivity-agent-2cqxl"
Jul 06 20:23:58.858907 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.858829    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kx5c4\" (UniqueName: \"kubernetes.io/projected/ed41c82c-bbcc-4385-8f89-f6ff340c2e36-kube-api-access-kx5c4\") pod \"node-ca-2hwd9\" (UID: \"ed41c82c-bbcc-4385-8f89-f6ff340c2e36\") " pod="openshift-image-registry/node-ca-2hwd9"
Jul 06 20:23:58.858907 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.858856    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/048dd56c-9989-44f2-a1d5-c3c50db6938c-system-cni-dir\") pod \"multus-additional-cni-plugins-b7wcr\" (UID: \"048dd56c-9989-44f2-a1d5-c3c50db6938c\") " pod="openshift-multus/multus-additional-cni-plugins-b7wcr"
Jul 06 20:23:58.858907 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.858890    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/64a3d8f1-f091-4ba3-9f90-1b77cfea3c71-os-release\") pod \"multus-gl5px\" (UID: \"64a3d8f1-f091-4ba3-9f90-1b77cfea3c71\") " pod="openshift-multus/multus-gl5px"
Jul 06 20:23:58.859159 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.858915    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sf5ls\" (UniqueName: \"kubernetes.io/projected/64a3d8f1-f091-4ba3-9f90-1b77cfea3c71-kube-api-access-sf5ls\") pod \"multus-gl5px\" (UID: \"64a3d8f1-f091-4ba3-9f90-1b77cfea3c71\") " pod="openshift-multus/multus-gl5px"
Jul 06 20:23:58.859159 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.858952    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e42d3c32-18ba-48dc-afb5-90ef7f0e2752-kubelet-config\") pod \"global-pull-secret-syncer-nzf7w\" (UID: \"e42d3c32-18ba-48dc-afb5-90ef7f0e2752\") " pod="kube-system/global-pull-secret-syncer-nzf7w"
Jul 06 20:23:58.859159 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.858978    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f48f979d-1c07-47ef-8144-488cad085d6e-tmp\") pod \"tuned-7pkrs\" (UID: \"f48f979d-1c07-47ef-8144-488cad085d6e\") " pod="openshift-cluster-node-tuning-operator/tuned-7pkrs"
Jul 06 20:23:58.859159 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.859005    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/64a3d8f1-f091-4ba3-9f90-1b77cfea3c71-multus-cni-dir\") pod \"multus-gl5px\" (UID: \"64a3d8f1-f091-4ba3-9f90-1b77cfea3c71\") " pod="openshift-multus/multus-gl5px"
Jul 06 20:23:58.859159 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.859029    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/64a3d8f1-f091-4ba3-9f90-1b77cfea3c71-host-var-lib-cni-multus\") pod \"multus-gl5px\" (UID: \"64a3d8f1-f091-4ba3-9f90-1b77cfea3c71\") " pod="openshift-multus/multus-gl5px"
Jul 06 20:23:58.859159 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.859052    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f48f979d-1c07-47ef-8144-488cad085d6e-host\") pod \"tuned-7pkrs\" (UID: \"f48f979d-1c07-47ef-8144-488cad085d6e\") " pod="openshift-cluster-node-tuning-operator/tuned-7pkrs"
Jul 06 20:23:58.859159 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.859077    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/048dd56c-9989-44f2-a1d5-c3c50db6938c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-b7wcr\" (UID: \"048dd56c-9989-44f2-a1d5-c3c50db6938c\") " pod="openshift-multus/multus-additional-cni-plugins-b7wcr"
Jul 06 20:23:58.859159 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.859104    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/66288c77-335a-4c17-9ba7-bee3609820cb-sys-fs\") pod \"aws-ebs-csi-driver-node-ls5cs\" (UID: \"66288c77-335a-4c17-9ba7-bee3609820cb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ls5cs"
Jul 06 20:23:58.859159 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.859128    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/64a3d8f1-f091-4ba3-9f90-1b77cfea3c71-cni-binary-copy\") pod \"multus-gl5px\" (UID: \"64a3d8f1-f091-4ba3-9f90-1b77cfea3c71\") " pod="openshift-multus/multus-gl5px"
Jul 06 20:23:58.859639 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.859154    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e42d3c32-18ba-48dc-afb5-90ef7f0e2752-original-pull-secret\") pod \"global-pull-secret-syncer-nzf7w\" (UID: \"e42d3c32-18ba-48dc-afb5-90ef7f0e2752\") " pod="kube-system/global-pull-secret-syncer-nzf7w"
Jul 06 20:23:58.859639 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.859192    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ed41c82c-bbcc-4385-8f89-f6ff340c2e36-serviceca\") pod \"node-ca-2hwd9\" (UID: \"ed41c82c-bbcc-4385-8f89-f6ff340c2e36\") " pod="openshift-image-registry/node-ca-2hwd9"
Jul 06 20:23:58.859639 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.859267    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/64a3d8f1-f091-4ba3-9f90-1b77cfea3c71-multus-socket-dir-parent\") pod \"multus-gl5px\" (UID: \"64a3d8f1-f091-4ba3-9f90-1b77cfea3c71\") " pod="openshift-multus/multus-gl5px"
Jul 06 20:23:58.859639 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.859295    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7f8a1214-4b3e-4de6-96fe-8f9be6cca7f7-host-slash\") pod \"iptables-alerter-48m7n\" (UID: \"7f8a1214-4b3e-4de6-96fe-8f9be6cca7f7\") " pod="openshift-network-operator/iptables-alerter-48m7n"
Jul 06 20:23:58.859639 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.859319    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f48f979d-1c07-47ef-8144-488cad085d6e-etc-modprobe-d\") pod \"tuned-7pkrs\" (UID: \"f48f979d-1c07-47ef-8144-488cad085d6e\") " pod="openshift-cluster-node-tuning-operator/tuned-7pkrs"
Jul 06 20:23:58.859639 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.859351    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ed41c82c-bbcc-4385-8f89-f6ff340c2e36-host\") pod \"node-ca-2hwd9\" (UID: \"ed41c82c-bbcc-4385-8f89-f6ff340c2e36\") " pod="openshift-image-registry/node-ca-2hwd9"
Jul 06 20:23:58.859639 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.859375    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/64a3d8f1-f091-4ba3-9f90-1b77cfea3c71-host-run-netns\") pod \"multus-gl5px\" (UID: \"64a3d8f1-f091-4ba3-9f90-1b77cfea3c71\") " pod="openshift-multus/multus-gl5px"
Jul 06 20:23:58.859639 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.859390    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/048dd56c-9989-44f2-a1d5-c3c50db6938c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-b7wcr\" (UID: \"048dd56c-9989-44f2-a1d5-c3c50db6938c\") " pod="openshift-multus/multus-additional-cni-plugins-b7wcr"
Jul 06 20:23:58.859639 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.859401    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lm8dh\" (UniqueName: \"kubernetes.io/projected/7f8a1214-4b3e-4de6-96fe-8f9be6cca7f7-kube-api-access-lm8dh\") pod \"iptables-alerter-48m7n\" (UID: \"7f8a1214-4b3e-4de6-96fe-8f9be6cca7f7\") " pod="openshift-network-operator/iptables-alerter-48m7n"
Jul 06 20:23:58.859639 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.859428    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f48f979d-1c07-47ef-8144-488cad085d6e-etc-kubernetes\") pod \"tuned-7pkrs\" (UID: \"f48f979d-1c07-47ef-8144-488cad085d6e\") " pod="openshift-cluster-node-tuning-operator/tuned-7pkrs"
Jul 06 20:23:58.859639 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.859470    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tk44x\" (UniqueName: \"kubernetes.io/projected/f48f979d-1c07-47ef-8144-488cad085d6e-kube-api-access-tk44x\") pod \"tuned-7pkrs\" (UID: \"f48f979d-1c07-47ef-8144-488cad085d6e\") " pod="openshift-cluster-node-tuning-operator/tuned-7pkrs"
Jul 06 20:23:58.859639 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.859505    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/048dd56c-9989-44f2-a1d5-c3c50db6938c-cnibin\") pod \"multus-additional-cni-plugins-b7wcr\" (UID: \"048dd56c-9989-44f2-a1d5-c3c50db6938c\") " pod="openshift-multus/multus-additional-cni-plugins-b7wcr"
Jul 06 20:23:58.859639 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.859530    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b20534ab-b8c8-4c81-a0f4-0f98a738896e-metrics-certs\") pod \"network-metrics-daemon-gvd8v\" (UID: \"b20534ab-b8c8-4c81-a0f4-0f98a738896e\") " pod="openshift-multus/network-metrics-daemon-gvd8v"
Jul 06 20:23:58.859639 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.859555    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/66288c77-335a-4c17-9ba7-bee3609820cb-registration-dir\") pod \"aws-ebs-csi-driver-node-ls5cs\" (UID: \"66288c77-335a-4c17-9ba7-bee3609820cb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ls5cs"
Jul 06 20:23:58.859639 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.859610    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/64a3d8f1-f091-4ba3-9f90-1b77cfea3c71-system-cni-dir\") pod \"multus-gl5px\" (UID: \"64a3d8f1-f091-4ba3-9f90-1b77cfea3c71\") " pod="openshift-multus/multus-gl5px"
Jul 06 20:23:58.860419 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.859665    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/64a3d8f1-f091-4ba3-9f90-1b77cfea3c71-cnibin\") pod \"multus-gl5px\" (UID: \"64a3d8f1-f091-4ba3-9f90-1b77cfea3c71\") " pod="openshift-multus/multus-gl5px"
Jul 06 20:23:58.860419 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.859707    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/64a3d8f1-f091-4ba3-9f90-1b77cfea3c71-host-var-lib-cni-bin\") pod \"multus-gl5px\" (UID: \"64a3d8f1-f091-4ba3-9f90-1b77cfea3c71\") " pod="openshift-multus/multus-gl5px"
Jul 06 20:23:58.860419 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.859726    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/048dd56c-9989-44f2-a1d5-c3c50db6938c-system-cni-dir\") pod \"multus-additional-cni-plugins-b7wcr\" (UID: \"048dd56c-9989-44f2-a1d5-c3c50db6938c\") " pod="openshift-multus/multus-additional-cni-plugins-b7wcr"
Jul 06 20:23:58.860419 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.859766    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/64a3d8f1-f091-4ba3-9f90-1b77cfea3c71-etc-kubernetes\") pod \"multus-gl5px\" (UID: \"64a3d8f1-f091-4ba3-9f90-1b77cfea3c71\") " pod="openshift-multus/multus-gl5px"
Jul 06 20:23:58.860419 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.859805    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e42d3c32-18ba-48dc-afb5-90ef7f0e2752-dbus\") pod \"global-pull-secret-syncer-nzf7w\" (UID: \"e42d3c32-18ba-48dc-afb5-90ef7f0e2752\") " pod="kube-system/global-pull-secret-syncer-nzf7w"
Jul 06 20:23:58.860419 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.859830    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/64a3d8f1-f091-4ba3-9f90-1b77cfea3c71-os-release\") pod \"multus-gl5px\" (UID: \"64a3d8f1-f091-4ba3-9f90-1b77cfea3c71\") " pod="openshift-multus/multus-gl5px"
Jul 06 20:23:58.860419 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.859831    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f48f979d-1c07-47ef-8144-488cad085d6e-etc-tuned\") pod \"tuned-7pkrs\" (UID: \"f48f979d-1c07-47ef-8144-488cad085d6e\") " pod="openshift-cluster-node-tuning-operator/tuned-7pkrs"
Jul 06 20:23:58.860419 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.859892    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/64a3d8f1-f091-4ba3-9f90-1b77cfea3c71-cnibin\") pod \"multus-gl5px\" (UID: \"64a3d8f1-f091-4ba3-9f90-1b77cfea3c71\") " pod="openshift-multus/multus-gl5px"
Jul 06 20:23:58.860419 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.859899    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/66288c77-335a-4c17-9ba7-bee3609820cb-socket-dir\") pod \"aws-ebs-csi-driver-node-ls5cs\" (UID: \"66288c77-335a-4c17-9ba7-bee3609820cb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ls5cs"
Jul 06 20:23:58.860419 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.859892    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/66288c77-335a-4c17-9ba7-bee3609820cb-sys-fs\") pod \"aws-ebs-csi-driver-node-ls5cs\" (UID: \"66288c77-335a-4c17-9ba7-bee3609820cb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ls5cs"
Jul 06 20:23:58.860419 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.859905    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/048dd56c-9989-44f2-a1d5-c3c50db6938c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-b7wcr\" (UID: \"048dd56c-9989-44f2-a1d5-c3c50db6938c\") " pod="openshift-multus/multus-additional-cni-plugins-b7wcr"
Jul 06 20:23:58.860419 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.859938    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/64a3d8f1-f091-4ba3-9f90-1b77cfea3c71-host-var-lib-kubelet\") pod \"multus-gl5px\" (UID: \"64a3d8f1-f091-4ba3-9f90-1b77cfea3c71\") " pod="openshift-multus/multus-gl5px"
Jul 06 20:23:58.860419 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.859966    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f48f979d-1c07-47ef-8144-488cad085d6e-var-lib-kubelet\") pod \"tuned-7pkrs\" (UID: \"f48f979d-1c07-47ef-8144-488cad085d6e\") " pod="openshift-cluster-node-tuning-operator/tuned-7pkrs"
Jul 06 20:23:58.860419 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.860022    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f48f979d-1c07-47ef-8144-488cad085d6e-var-lib-kubelet\") pod \"tuned-7pkrs\" (UID: \"f48f979d-1c07-47ef-8144-488cad085d6e\") " pod="openshift-cluster-node-tuning-operator/tuned-7pkrs"
Jul 06 20:23:58.860419 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.860031    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/66288c77-335a-4c17-9ba7-bee3609820cb-socket-dir\") pod \"aws-ebs-csi-driver-node-ls5cs\" (UID: \"66288c77-335a-4c17-9ba7-bee3609820cb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ls5cs"
Jul 06 20:23:58.860419 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.860089    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/64a3d8f1-f091-4ba3-9f90-1b77cfea3c71-host-var-lib-cni-bin\") pod \"multus-gl5px\" (UID: \"64a3d8f1-f091-4ba3-9f90-1b77cfea3c71\") " pod="openshift-multus/multus-gl5px"
Jul 06 20:23:58.860419 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.860129    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/64a3d8f1-f091-4ba3-9f90-1b77cfea3c71-etc-kubernetes\") pod \"multus-gl5px\" (UID: \"64a3d8f1-f091-4ba3-9f90-1b77cfea3c71\") " pod="openshift-multus/multus-gl5px"
Jul 06 20:23:58.860419 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.860349    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e42d3c32-18ba-48dc-afb5-90ef7f0e2752-dbus\") pod \"global-pull-secret-syncer-nzf7w\" (UID: \"e42d3c32-18ba-48dc-afb5-90ef7f0e2752\") " pod="kube-system/global-pull-secret-syncer-nzf7w"
Jul 06 20:23:58.861324 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.859627    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/66288c77-335a-4c17-9ba7-bee3609820cb-registration-dir\") pod \"aws-ebs-csi-driver-node-ls5cs\" (UID: \"66288c77-335a-4c17-9ba7-bee3609820cb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ls5cs"
Jul 06 20:23:58.861324 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.860429    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/64a3d8f1-f091-4ba3-9f90-1b77cfea3c71-system-cni-dir\") pod \"multus-gl5px\" (UID: \"64a3d8f1-f091-4ba3-9f90-1b77cfea3c71\") " pod="openshift-multus/multus-gl5px"
Jul 06 20:23:58.861324 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.860447    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/64a3d8f1-f091-4ba3-9f90-1b77cfea3c71-multus-cni-dir\") pod \"multus-gl5px\" (UID: \"64a3d8f1-f091-4ba3-9f90-1b77cfea3c71\") " pod="openshift-multus/multus-gl5px"
Jul 06 20:23:58.861324 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.860483    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f48f979d-1c07-47ef-8144-488cad085d6e-host\") pod \"tuned-7pkrs\" (UID: \"f48f979d-1c07-47ef-8144-488cad085d6e\") " pod="openshift-cluster-node-tuning-operator/tuned-7pkrs"
Jul 06 20:23:58.861324 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.860495    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/64a3d8f1-f091-4ba3-9f90-1b77cfea3c71-host-var-lib-cni-multus\") pod \"multus-gl5px\" (UID: \"64a3d8f1-f091-4ba3-9f90-1b77cfea3c71\") " pod="openshift-multus/multus-gl5px"
Jul 06 20:23:58.861324 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.860507    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/64a3d8f1-f091-4ba3-9f90-1b77cfea3c71-cni-binary-copy\") pod \"multus-gl5px\" (UID: \"64a3d8f1-f091-4ba3-9f90-1b77cfea3c71\") " pod="openshift-multus/multus-gl5px"
Jul 06 20:23:58.861324 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.860535    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/64a3d8f1-f091-4ba3-9f90-1b77cfea3c71-host-var-lib-kubelet\") pod \"multus-gl5px\" (UID: \"64a3d8f1-f091-4ba3-9f90-1b77cfea3c71\") " pod="openshift-multus/multus-gl5px"
Jul 06 20:23:58.861324 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.860589    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ed41c82c-bbcc-4385-8f89-f6ff340c2e36-host\") pod \"node-ca-2hwd9\" (UID: \"ed41c82c-bbcc-4385-8f89-f6ff340c2e36\") " pod="openshift-image-registry/node-ca-2hwd9"
Jul 06 20:23:58.861324 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:23:58.860630    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 06 20:23:58.861324 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.860640    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/64a3d8f1-f091-4ba3-9f90-1b77cfea3c71-host-run-netns\") pod \"multus-gl5px\" (UID: \"64a3d8f1-f091-4ba3-9f90-1b77cfea3c71\") " pod="openshift-multus/multus-gl5px"
Jul 06 20:23:58.861324 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.860650    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7f8a1214-4b3e-4de6-96fe-8f9be6cca7f7-host-slash\") pod \"iptables-alerter-48m7n\" (UID: \"7f8a1214-4b3e-4de6-96fe-8f9be6cca7f7\") " pod="openshift-network-operator/iptables-alerter-48m7n"
Jul 06 20:23:58.861324 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:23:58.860713    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e42d3c32-18ba-48dc-afb5-90ef7f0e2752-original-pull-secret podName:e42d3c32-18ba-48dc-afb5-90ef7f0e2752 nodeName:}" failed. No retries permitted until 2026-07-06 20:23:59.360675661 +0000 UTC m=+3.076786320 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e42d3c32-18ba-48dc-afb5-90ef7f0e2752-original-pull-secret") pod "global-pull-secret-syncer-nzf7w" (UID: "e42d3c32-18ba-48dc-afb5-90ef7f0e2752") : object "kube-system"/"original-pull-secret" not registered
Jul 06 20:23:58.861324 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.860730    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/64a3d8f1-f091-4ba3-9f90-1b77cfea3c71-multus-socket-dir-parent\") pod \"multus-gl5px\" (UID: \"64a3d8f1-f091-4ba3-9f90-1b77cfea3c71\") " pod="openshift-multus/multus-gl5px"
Jul 06 20:23:58.861324 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.860748    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/048dd56c-9989-44f2-a1d5-c3c50db6938c-cnibin\") pod \"multus-additional-cni-plugins-b7wcr\" (UID: \"048dd56c-9989-44f2-a1d5-c3c50db6938c\") " pod="openshift-multus/multus-additional-cni-plugins-b7wcr"
Jul 06 20:23:58.861324 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.860790    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/66288c77-335a-4c17-9ba7-bee3609820cb-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ls5cs\" (UID: \"66288c77-335a-4c17-9ba7-bee3609820cb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ls5cs"
Jul 06 20:23:58.861324 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.860812    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f48f979d-1c07-47ef-8144-488cad085d6e-etc-kubernetes\") pod \"tuned-7pkrs\" (UID: \"f48f979d-1c07-47ef-8144-488cad085d6e\") " pod="openshift-cluster-node-tuning-operator/tuned-7pkrs"
Jul 06 20:23:58.861324 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.860922    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f48f979d-1c07-47ef-8144-488cad085d6e-etc-modprobe-d\") pod \"tuned-7pkrs\" (UID: \"f48f979d-1c07-47ef-8144-488cad085d6e\") " pod="openshift-cluster-node-tuning-operator/tuned-7pkrs"
Jul 06 20:23:58.862226 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.860734    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/66288c77-335a-4c17-9ba7-bee3609820cb-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ls5cs\" (UID: \"66288c77-335a-4c17-9ba7-bee3609820cb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ls5cs"
Jul 06 20:23:58.862226 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:23:58.860969    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 20:23:58.862226 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.860973    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/66288c77-335a-4c17-9ba7-bee3609820cb-device-dir\") pod \"aws-ebs-csi-driver-node-ls5cs\" (UID: \"66288c77-335a-4c17-9ba7-bee3609820cb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ls5cs"
Jul 06 20:23:58.862226 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.861002    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/64a3d8f1-f091-4ba3-9f90-1b77cfea3c71-host-run-k8s-cni-cncf-io\") pod \"multus-gl5px\" (UID: \"64a3d8f1-f091-4ba3-9f90-1b77cfea3c71\") " pod="openshift-multus/multus-gl5px"
Jul 06 20:23:58.862226 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.861003    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ed41c82c-bbcc-4385-8f89-f6ff340c2e36-serviceca\") pod \"node-ca-2hwd9\" (UID: \"ed41c82c-bbcc-4385-8f89-f6ff340c2e36\") " pod="openshift-image-registry/node-ca-2hwd9"
Jul 06 20:23:58.862226 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.861033    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f48f979d-1c07-47ef-8144-488cad085d6e-sys\") pod \"tuned-7pkrs\" (UID: \"f48f979d-1c07-47ef-8144-488cad085d6e\") " pod="openshift-cluster-node-tuning-operator/tuned-7pkrs"
Jul 06 20:23:58.862226 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.861067    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/048dd56c-9989-44f2-a1d5-c3c50db6938c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-b7wcr\" (UID: \"048dd56c-9989-44f2-a1d5-c3c50db6938c\") " pod="openshift-multus/multus-additional-cni-plugins-b7wcr"
Jul 06 20:23:58.862226 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.861081    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/66288c77-335a-4c17-9ba7-bee3609820cb-device-dir\") pod \"aws-ebs-csi-driver-node-ls5cs\" (UID: \"66288c77-335a-4c17-9ba7-bee3609820cb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ls5cs"
Jul 06 20:23:58.862226 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:23:58.861086    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b20534ab-b8c8-4c81-a0f4-0f98a738896e-metrics-certs podName:b20534ab-b8c8-4c81-a0f4-0f98a738896e nodeName:}" failed. No retries permitted until 2026-07-06 20:23:59.36106881 +0000 UTC m=+3.077179467 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b20534ab-b8c8-4c81-a0f4-0f98a738896e-metrics-certs") pod "network-metrics-daemon-gvd8v" (UID: "b20534ab-b8c8-4c81-a0f4-0f98a738896e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 20:23:58.862226 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.861107    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/64a3d8f1-f091-4ba3-9f90-1b77cfea3c71-host-run-k8s-cni-cncf-io\") pod \"multus-gl5px\" (UID: \"64a3d8f1-f091-4ba3-9f90-1b77cfea3c71\") " pod="openshift-multus/multus-gl5px"
Jul 06 20:23:58.862226 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.861134    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e42d3c32-18ba-48dc-afb5-90ef7f0e2752-kubelet-config\") pod \"global-pull-secret-syncer-nzf7w\" (UID: \"e42d3c32-18ba-48dc-afb5-90ef7f0e2752\") " pod="kube-system/global-pull-secret-syncer-nzf7w"
Jul 06 20:23:58.862226 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.861150    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/64a3d8f1-f091-4ba3-9f90-1b77cfea3c71-hostroot\") pod \"multus-gl5px\" (UID: \"64a3d8f1-f091-4ba3-9f90-1b77cfea3c71\") " pod="openshift-multus/multus-gl5px"
Jul 06 20:23:58.862226 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.861185    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/64a3d8f1-f091-4ba3-9f90-1b77cfea3c71-multus-conf-dir\") pod \"multus-gl5px\" (UID: \"64a3d8f1-f091-4ba3-9f90-1b77cfea3c71\") " pod="openshift-multus/multus-gl5px"
Jul 06 20:23:58.862226 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.861210    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f48f979d-1c07-47ef-8144-488cad085d6e-lib-modules\") pod \"tuned-7pkrs\" (UID: \"f48f979d-1c07-47ef-8144-488cad085d6e\") " pod="openshift-cluster-node-tuning-operator/tuned-7pkrs"
Jul 06 20:23:58.862226 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.861152    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f48f979d-1c07-47ef-8144-488cad085d6e-sys\") pod \"tuned-7pkrs\" (UID: \"f48f979d-1c07-47ef-8144-488cad085d6e\") " pod="openshift-cluster-node-tuning-operator/tuned-7pkrs"
Jul 06 20:23:58.862226 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.861217    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/64a3d8f1-f091-4ba3-9f90-1b77cfea3c71-multus-conf-dir\") pod \"multus-gl5px\" (UID: \"64a3d8f1-f091-4ba3-9f90-1b77cfea3c71\") " pod="openshift-multus/multus-gl5px"
Jul 06 20:23:58.862226 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.861185    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/64a3d8f1-f091-4ba3-9f90-1b77cfea3c71-hostroot\") pod \"multus-gl5px\" (UID: \"64a3d8f1-f091-4ba3-9f90-1b77cfea3c71\") " pod="openshift-multus/multus-gl5px"
Jul 06 20:23:58.863115 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.861210    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/048dd56c-9989-44f2-a1d5-c3c50db6938c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-b7wcr\" (UID: \"048dd56c-9989-44f2-a1d5-c3c50db6938c\") " pod="openshift-multus/multus-additional-cni-plugins-b7wcr"
Jul 06 20:23:58.863115 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.861260    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/66288c77-335a-4c17-9ba7-bee3609820cb-etc-selinux\") pod \"aws-ebs-csi-driver-node-ls5cs\" (UID: \"66288c77-335a-4c17-9ba7-bee3609820cb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ls5cs"
Jul 06 20:23:58.863115 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.861290    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f48f979d-1c07-47ef-8144-488cad085d6e-etc-sysctl-conf\") pod \"tuned-7pkrs\" (UID: \"f48f979d-1c07-47ef-8144-488cad085d6e\") " pod="openshift-cluster-node-tuning-operator/tuned-7pkrs"
Jul 06 20:23:58.863115 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.861316    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/048dd56c-9989-44f2-a1d5-c3c50db6938c-cni-binary-copy\") pod \"multus-additional-cni-plugins-b7wcr\" (UID: \"048dd56c-9989-44f2-a1d5-c3c50db6938c\") " pod="openshift-multus/multus-additional-cni-plugins-b7wcr"
Jul 06 20:23:58.863115 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.861341    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f48f979d-1c07-47ef-8144-488cad085d6e-lib-modules\") pod \"tuned-7pkrs\" (UID: \"f48f979d-1c07-47ef-8144-488cad085d6e\") " pod="openshift-cluster-node-tuning-operator/tuned-7pkrs"
Jul 06 20:23:58.863115 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.861344    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kqkd\" (UniqueName: \"kubernetes.io/projected/66288c77-335a-4c17-9ba7-bee3609820cb-kube-api-access-4kqkd\") pod \"aws-ebs-csi-driver-node-ls5cs\" (UID: \"66288c77-335a-4c17-9ba7-bee3609820cb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ls5cs"
Jul 06 20:23:58.863115 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.861383    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f48f979d-1c07-47ef-8144-488cad085d6e-etc-sysctl-d\") pod \"tuned-7pkrs\" (UID: \"f48f979d-1c07-47ef-8144-488cad085d6e\") " pod="openshift-cluster-node-tuning-operator/tuned-7pkrs"
Jul 06 20:23:58.863115 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.861390    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/66288c77-335a-4c17-9ba7-bee3609820cb-etc-selinux\") pod \"aws-ebs-csi-driver-node-ls5cs\" (UID: \"66288c77-335a-4c17-9ba7-bee3609820cb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ls5cs"
Jul 06 20:23:58.863115 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.861410    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f48f979d-1c07-47ef-8144-488cad085d6e-run\") pod \"tuned-7pkrs\" (UID: \"f48f979d-1c07-47ef-8144-488cad085d6e\") " pod="openshift-cluster-node-tuning-operator/tuned-7pkrs"
Jul 06 20:23:58.863115 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.861436    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5xth\" (UniqueName: \"kubernetes.io/projected/b20534ab-b8c8-4c81-a0f4-0f98a738896e-kube-api-access-x5xth\") pod \"network-metrics-daemon-gvd8v\" (UID: \"b20534ab-b8c8-4c81-a0f4-0f98a738896e\") " pod="openshift-multus/network-metrics-daemon-gvd8v"
Jul 06 20:23:58.863115 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.861478    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/64a3d8f1-f091-4ba3-9f90-1b77cfea3c71-multus-daemon-config\") pod \"multus-gl5px\" (UID: \"64a3d8f1-f091-4ba3-9f90-1b77cfea3c71\") " pod="openshift-multus/multus-gl5px"
Jul 06 20:23:58.863115 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.861503    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/64a3d8f1-f091-4ba3-9f90-1b77cfea3c71-host-run-multus-certs\") pod \"multus-gl5px\" (UID: \"64a3d8f1-f091-4ba3-9f90-1b77cfea3c71\") " pod="openshift-multus/multus-gl5px"
Jul 06 20:23:58.863115 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.861528    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7f8a1214-4b3e-4de6-96fe-8f9be6cca7f7-iptables-alerter-script\") pod \"iptables-alerter-48m7n\" (UID: \"7f8a1214-4b3e-4de6-96fe-8f9be6cca7f7\") " pod="openshift-network-operator/iptables-alerter-48m7n"
Jul 06 20:23:58.863115 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.861551    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f48f979d-1c07-47ef-8144-488cad085d6e-etc-systemd\") pod \"tuned-7pkrs\" (UID: \"f48f979d-1c07-47ef-8144-488cad085d6e\") " pod="openshift-cluster-node-tuning-operator/tuned-7pkrs"
Jul 06 20:23:58.863115 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.861599    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9wr4\" (UniqueName: \"kubernetes.io/projected/d2969391-caa9-4c3d-932c-3727d57b6ce8-kube-api-access-m9wr4\") pod \"network-check-target-tc5ls\" (UID: \"d2969391-caa9-4c3d-932c-3727d57b6ce8\") " pod="openshift-network-diagnostics/network-check-target-tc5ls"
Jul 06 20:23:58.863115 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.861624    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwgtw\" (UniqueName: \"kubernetes.io/projected/048dd56c-9989-44f2-a1d5-c3c50db6938c-kube-api-access-gwgtw\") pod \"multus-additional-cni-plugins-b7wcr\" (UID: \"048dd56c-9989-44f2-a1d5-c3c50db6938c\") " pod="openshift-multus/multus-additional-cni-plugins-b7wcr"
Jul 06 20:23:58.863115 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.861677    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/408f722a-5e7e-4bad-a86e-d461b5fed747-konnectivity-ca\") pod \"konnectivity-agent-2cqxl\" (UID: \"408f722a-5e7e-4bad-a86e-d461b5fed747\") " pod="kube-system/konnectivity-agent-2cqxl"
Jul 06 20:23:58.864038 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.861704    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f48f979d-1c07-47ef-8144-488cad085d6e-etc-sysconfig\") pod \"tuned-7pkrs\" (UID: \"f48f979d-1c07-47ef-8144-488cad085d6e\") " pod="openshift-cluster-node-tuning-operator/tuned-7pkrs"
Jul 06 20:23:58.864038 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.861740    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/048dd56c-9989-44f2-a1d5-c3c50db6938c-os-release\") pod \"multus-additional-cni-plugins-b7wcr\" (UID: \"048dd56c-9989-44f2-a1d5-c3c50db6938c\") " pod="openshift-multus/multus-additional-cni-plugins-b7wcr"
Jul 06 20:23:58.864038 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.861892    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/048dd56c-9989-44f2-a1d5-c3c50db6938c-cni-binary-copy\") pod \"multus-additional-cni-plugins-b7wcr\" (UID: \"048dd56c-9989-44f2-a1d5-c3c50db6938c\") " pod="openshift-multus/multus-additional-cni-plugins-b7wcr"
Jul 06 20:23:58.864038 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.861914    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/048dd56c-9989-44f2-a1d5-c3c50db6938c-os-release\") pod \"multus-additional-cni-plugins-b7wcr\" (UID: \"048dd56c-9989-44f2-a1d5-c3c50db6938c\") " pod="openshift-multus/multus-additional-cni-plugins-b7wcr"
Jul 06 20:23:58.864038 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.861974    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f48f979d-1c07-47ef-8144-488cad085d6e-etc-sysctl-conf\") pod \"tuned-7pkrs\" (UID: \"f48f979d-1c07-47ef-8144-488cad085d6e\") " pod="openshift-cluster-node-tuning-operator/tuned-7pkrs"
Jul 06 20:23:58.864038 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.862045    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f48f979d-1c07-47ef-8144-488cad085d6e-run\") pod \"tuned-7pkrs\" (UID: \"f48f979d-1c07-47ef-8144-488cad085d6e\") " pod="openshift-cluster-node-tuning-operator/tuned-7pkrs"
Jul 06 20:23:58.864038 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.862150    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f48f979d-1c07-47ef-8144-488cad085d6e-etc-sysctl-d\") pod \"tuned-7pkrs\" (UID: \"f48f979d-1c07-47ef-8144-488cad085d6e\") " pod="openshift-cluster-node-tuning-operator/tuned-7pkrs"
Jul 06 20:23:58.864038 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.862195    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/64a3d8f1-f091-4ba3-9f90-1b77cfea3c71-multus-daemon-config\") pod \"multus-gl5px\" (UID: \"64a3d8f1-f091-4ba3-9f90-1b77cfea3c71\") " pod="openshift-multus/multus-gl5px"
Jul 06 20:23:58.864038 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.862246    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f48f979d-1c07-47ef-8144-488cad085d6e-etc-systemd\") pod \"tuned-7pkrs\" (UID: \"f48f979d-1c07-47ef-8144-488cad085d6e\") " pod="openshift-cluster-node-tuning-operator/tuned-7pkrs"
Jul 06 20:23:58.864038 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.862304    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/64a3d8f1-f091-4ba3-9f90-1b77cfea3c71-host-run-multus-certs\") pod \"multus-gl5px\" (UID: \"64a3d8f1-f091-4ba3-9f90-1b77cfea3c71\") " pod="openshift-multus/multus-gl5px"
Jul 06 20:23:58.864038 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.862360    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f48f979d-1c07-47ef-8144-488cad085d6e-etc-sysconfig\") pod \"tuned-7pkrs\" (UID: \"f48f979d-1c07-47ef-8144-488cad085d6e\") " pod="openshift-cluster-node-tuning-operator/tuned-7pkrs"
Jul 06 20:23:58.864038 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.862758    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/408f722a-5e7e-4bad-a86e-d461b5fed747-konnectivity-ca\") pod \"konnectivity-agent-2cqxl\" (UID: \"408f722a-5e7e-4bad-a86e-d461b5fed747\") " pod="kube-system/konnectivity-agent-2cqxl"
Jul 06 20:23:58.864038 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.862780    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7f8a1214-4b3e-4de6-96fe-8f9be6cca7f7-iptables-alerter-script\") pod \"iptables-alerter-48m7n\" (UID: \"7f8a1214-4b3e-4de6-96fe-8f9be6cca7f7\") " pod="openshift-network-operator/iptables-alerter-48m7n"
Jul 06 20:23:58.864038 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.863204    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f48f979d-1c07-47ef-8144-488cad085d6e-tmp\") pod \"tuned-7pkrs\" (UID: \"f48f979d-1c07-47ef-8144-488cad085d6e\") " pod="openshift-cluster-node-tuning-operator/tuned-7pkrs"
Jul 06 20:23:58.864038 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.863485    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f48f979d-1c07-47ef-8144-488cad085d6e-etc-tuned\") pod \"tuned-7pkrs\" (UID: \"f48f979d-1c07-47ef-8144-488cad085d6e\") " pod="openshift-cluster-node-tuning-operator/tuned-7pkrs"
Jul 06 20:23:58.865122 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.865098    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/408f722a-5e7e-4bad-a86e-d461b5fed747-agent-certs\") pod \"konnectivity-agent-2cqxl\" (UID: \"408f722a-5e7e-4bad-a86e-d461b5fed747\") " pod="kube-system/konnectivity-agent-2cqxl"
Jul 06 20:23:58.869618 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:23:58.869480    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 06 20:23:58.869618 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:23:58.869502    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 06 20:23:58.869618 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:23:58.869516    2563 projected.go:196] Error preparing data for projected volume kube-api-access-m9wr4 for pod openshift-network-diagnostics/network-check-target-tc5ls: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 20:23:58.869618 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:23:58.869583    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d2969391-caa9-4c3d-932c-3727d57b6ce8-kube-api-access-m9wr4 podName:d2969391-caa9-4c3d-932c-3727d57b6ce8 nodeName:}" failed. No retries permitted until 2026-07-06 20:23:59.36956485 +0000 UTC m=+3.085675518 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-m9wr4" (UniqueName: "kubernetes.io/projected/d2969391-caa9-4c3d-932c-3727d57b6ce8-kube-api-access-m9wr4") pod "network-check-target-tc5ls" (UID: "d2969391-caa9-4c3d-932c-3727d57b6ce8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 20:23:58.869933 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.869860    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sf5ls\" (UniqueName: \"kubernetes.io/projected/64a3d8f1-f091-4ba3-9f90-1b77cfea3c71-kube-api-access-sf5ls\") pod \"multus-gl5px\" (UID: \"64a3d8f1-f091-4ba3-9f90-1b77cfea3c71\") " pod="openshift-multus/multus-gl5px"
Jul 06 20:23:58.870794 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.870445    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kx5c4\" (UniqueName: \"kubernetes.io/projected/ed41c82c-bbcc-4385-8f89-f6ff340c2e36-kube-api-access-kx5c4\") pod \"node-ca-2hwd9\" (UID: \"ed41c82c-bbcc-4385-8f89-f6ff340c2e36\") " pod="openshift-image-registry/node-ca-2hwd9"
Jul 06 20:23:58.876502 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.872779    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lm8dh\" (UniqueName: \"kubernetes.io/projected/7f8a1214-4b3e-4de6-96fe-8f9be6cca7f7-kube-api-access-lm8dh\") pod \"iptables-alerter-48m7n\" (UID: \"7f8a1214-4b3e-4de6-96fe-8f9be6cca7f7\") " pod="openshift-network-operator/iptables-alerter-48m7n"
Jul 06 20:23:58.877902 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.877846    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5xth\" (UniqueName: \"kubernetes.io/projected/b20534ab-b8c8-4c81-a0f4-0f98a738896e-kube-api-access-x5xth\") pod \"network-metrics-daemon-gvd8v\" (UID: \"b20534ab-b8c8-4c81-a0f4-0f98a738896e\") " pod="openshift-multus/network-metrics-daemon-gvd8v"
Jul 06 20:23:58.878494 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.878445    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tk44x\" (UniqueName: \"kubernetes.io/projected/f48f979d-1c07-47ef-8144-488cad085d6e-kube-api-access-tk44x\") pod \"tuned-7pkrs\" (UID: \"f48f979d-1c07-47ef-8144-488cad085d6e\") " pod="openshift-cluster-node-tuning-operator/tuned-7pkrs"
Jul 06 20:23:58.879448 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.879422    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwgtw\" (UniqueName: \"kubernetes.io/projected/048dd56c-9989-44f2-a1d5-c3c50db6938c-kube-api-access-gwgtw\") pod \"multus-additional-cni-plugins-b7wcr\" (UID: \"048dd56c-9989-44f2-a1d5-c3c50db6938c\") " pod="openshift-multus/multus-additional-cni-plugins-b7wcr"
Jul 06 20:23:58.880111 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.880088    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kqkd\" (UniqueName: \"kubernetes.io/projected/66288c77-335a-4c17-9ba7-bee3609820cb-kube-api-access-4kqkd\") pod \"aws-ebs-csi-driver-node-ls5cs\" (UID: \"66288c77-335a-4c17-9ba7-bee3609820cb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ls5cs"
Jul 06 20:23:58.955833 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.955795    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-5tk9w"
Jul 06 20:23:58.964441 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.964375    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-n9568"
Jul 06 20:23:58.975172 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.975147    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-7pkrs"
Jul 06 20:23:58.983930 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.983907    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-b7wcr"
Jul 06 20:23:58.993580 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:58.993545    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-2cqxl"
Jul 06 20:23:59.000312 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:59.000289    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-2hwd9"
Jul 06 20:23:59.013953 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:59.013927    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-gl5px"
Jul 06 20:23:59.019698 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:59.019677    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ls5cs"
Jul 06 20:23:59.027457 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:59.027434    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-48m7n"
Jul 06 20:23:59.366224 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:59.366142    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e42d3c32-18ba-48dc-afb5-90ef7f0e2752-original-pull-secret\") pod \"global-pull-secret-syncer-nzf7w\" (UID: \"e42d3c32-18ba-48dc-afb5-90ef7f0e2752\") " pod="kube-system/global-pull-secret-syncer-nzf7w"
Jul 06 20:23:59.366224 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:59.366190    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b20534ab-b8c8-4c81-a0f4-0f98a738896e-metrics-certs\") pod \"network-metrics-daemon-gvd8v\" (UID: \"b20534ab-b8c8-4c81-a0f4-0f98a738896e\") " pod="openshift-multus/network-metrics-daemon-gvd8v"
Jul 06 20:23:59.366433 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:23:59.366302    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 06 20:23:59.366433 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:23:59.366369    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e42d3c32-18ba-48dc-afb5-90ef7f0e2752-original-pull-secret podName:e42d3c32-18ba-48dc-afb5-90ef7f0e2752 nodeName:}" failed. No retries permitted until 2026-07-06 20:24:00.366353505 +0000 UTC m=+4.082464146 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e42d3c32-18ba-48dc-afb5-90ef7f0e2752-original-pull-secret") pod "global-pull-secret-syncer-nzf7w" (UID: "e42d3c32-18ba-48dc-afb5-90ef7f0e2752") : object "kube-system"/"original-pull-secret" not registered
Jul 06 20:23:59.366433 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:23:59.366305    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 20:23:59.366537 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:23:59.366446    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b20534ab-b8c8-4c81-a0f4-0f98a738896e-metrics-certs podName:b20534ab-b8c8-4c81-a0f4-0f98a738896e nodeName:}" failed. No retries permitted until 2026-07-06 20:24:00.366430883 +0000 UTC m=+4.082541524 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b20534ab-b8c8-4c81-a0f4-0f98a738896e-metrics-certs") pod "network-metrics-daemon-gvd8v" (UID: "b20534ab-b8c8-4c81-a0f4-0f98a738896e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 20:23:59.467838 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:59.467802    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9wr4\" (UniqueName: \"kubernetes.io/projected/d2969391-caa9-4c3d-932c-3727d57b6ce8-kube-api-access-m9wr4\") pod \"network-check-target-tc5ls\" (UID: \"d2969391-caa9-4c3d-932c-3727d57b6ce8\") " pod="openshift-network-diagnostics/network-check-target-tc5ls"
Jul 06 20:23:59.467999 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:23:59.467932    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 06 20:23:59.467999 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:23:59.467948    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 06 20:23:59.467999 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:23:59.467957    2563 projected.go:196] Error preparing data for projected volume kube-api-access-m9wr4 for pod openshift-network-diagnostics/network-check-target-tc5ls: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 20:23:59.468132 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:23:59.468020    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d2969391-caa9-4c3d-932c-3727d57b6ce8-kube-api-access-m9wr4 podName:d2969391-caa9-4c3d-932c-3727d57b6ce8 nodeName:}" failed. No retries permitted until 2026-07-06 20:24:00.46800603 +0000 UTC m=+4.184116670 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-m9wr4" (UniqueName: "kubernetes.io/projected/d2969391-caa9-4c3d-932c-3727d57b6ce8-kube-api-access-m9wr4") pod "network-check-target-tc5ls" (UID: "d2969391-caa9-4c3d-932c-3727d57b6ce8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 20:23:59.725122 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:59.725029    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-05 20:18:59 +0000 UTC" deadline="2028-03-26 16:47:39.251107894 +0000 UTC"
Jul 06 20:23:59.725122 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:23:59.725062    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15092h23m39.52604936s"
Jul 06 20:24:00.047653 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:24:00.047619    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64a3d8f1_f091_4ba3_9f90_1b77cfea3c71.slice/crio-0d36dc6a26012e38bb55d6171c43254065bc8bbebb24d41c8897d8b859b13f0c WatchSource:0}: Error finding container 0d36dc6a26012e38bb55d6171c43254065bc8bbebb24d41c8897d8b859b13f0c: Status 404 returned error can't find the container with id 0d36dc6a26012e38bb55d6171c43254065bc8bbebb24d41c8897d8b859b13f0c
Jul 06 20:24:00.051696 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:24:00.051666    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod408f722a_5e7e_4bad_a86e_d461b5fed747.slice/crio-002a998b0c81f1e882286b40d3d4338fa4d75fa3f46b21929e2926b8597566df WatchSource:0}: Error finding container 002a998b0c81f1e882286b40d3d4338fa4d75fa3f46b21929e2926b8597566df: Status 404 returned error can't find the container with id 002a998b0c81f1e882286b40d3d4338fa4d75fa3f46b21929e2926b8597566df
Jul 06 20:24:00.054257 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:24:00.054226    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66288c77_335a_4c17_9ba7_bee3609820cb.slice/crio-261909b9019d66089020bff965387d00558341893f7bc40372f2f2e0900974b6 WatchSource:0}: Error finding container 261909b9019d66089020bff965387d00558341893f7bc40372f2f2e0900974b6: Status 404 returned error can't find the container with id 261909b9019d66089020bff965387d00558341893f7bc40372f2f2e0900974b6
Jul 06 20:24:00.055610 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:24:00.055581    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded41c82c_bbcc_4385_8f89_f6ff340c2e36.slice/crio-c820514c2b7827cea236bb37d8eeb2f0a727a3777a5956f84ea9447d6a0a6069 WatchSource:0}: Error finding container c820514c2b7827cea236bb37d8eeb2f0a727a3777a5956f84ea9447d6a0a6069: Status 404 returned error can't find the container with id c820514c2b7827cea236bb37d8eeb2f0a727a3777a5956f84ea9447d6a0a6069
Jul 06 20:24:00.056304 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:24:00.056279    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod048dd56c_9989_44f2_a1d5_c3c50db6938c.slice/crio-2c4fd307dcf304aff922db9d78cc12f1efdad1cf6bea0433774c92eb4879abde WatchSource:0}: Error finding container 2c4fd307dcf304aff922db9d78cc12f1efdad1cf6bea0433774c92eb4879abde: Status 404 returned error can't find the container with id 2c4fd307dcf304aff922db9d78cc12f1efdad1cf6bea0433774c92eb4879abde
Jul 06 20:24:00.057995 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:24:00.057822    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1816531c_ac4a_4d49_96ac_234daa36ea91.slice/crio-ee400fdb2fc840844f8f10c91b940d16ae03ebeb5798bad8a4dac0e5b8c43c3d WatchSource:0}: Error finding container ee400fdb2fc840844f8f10c91b940d16ae03ebeb5798bad8a4dac0e5b8c43c3d: Status 404 returned error can't find the container with id ee400fdb2fc840844f8f10c91b940d16ae03ebeb5798bad8a4dac0e5b8c43c3d
Jul 06 20:24:00.059511 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:24:00.059313    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f8a1214_4b3e_4de6_96fe_8f9be6cca7f7.slice/crio-3c31ff108e4861a56ecb5ac94fdfa3192c6cb547a2fc6cf3eb221b244f45c9f6 WatchSource:0}: Error finding container 3c31ff108e4861a56ecb5ac94fdfa3192c6cb547a2fc6cf3eb221b244f45c9f6: Status 404 returned error can't find the container with id 3c31ff108e4861a56ecb5ac94fdfa3192c6cb547a2fc6cf3eb221b244f45c9f6
Jul 06 20:24:00.061102 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:24:00.060722    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf48f979d_1c07_47ef_8144_488cad085d6e.slice/crio-f9cfad28a382001f74360872368989b6b42c7372d770b2287dac14c9b609721e WatchSource:0}: Error finding container f9cfad28a382001f74360872368989b6b42c7372d770b2287dac14c9b609721e: Status 404 returned error can't find the container with id f9cfad28a382001f74360872368989b6b42c7372d770b2287dac14c9b609721e
Jul 06 20:24:00.062405 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:24:00.062375    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53799812_6492_4e52_94be_ce6e529cfc88.slice/crio-09edead3b454edc627311f59e1dd6f4682c6a20f4e5e8ae6b21221708c0807f9 WatchSource:0}: Error finding container 09edead3b454edc627311f59e1dd6f4682c6a20f4e5e8ae6b21221708c0807f9: Status 404 returned error can't find the container with id 09edead3b454edc627311f59e1dd6f4682c6a20f4e5e8ae6b21221708c0807f9
Jul 06 20:24:00.375668 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:00.375465    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e42d3c32-18ba-48dc-afb5-90ef7f0e2752-original-pull-secret\") pod \"global-pull-secret-syncer-nzf7w\" (UID: \"e42d3c32-18ba-48dc-afb5-90ef7f0e2752\") " pod="kube-system/global-pull-secret-syncer-nzf7w"
Jul 06 20:24:00.375845 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:24:00.375681    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 06 20:24:00.375845 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:00.375696    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b20534ab-b8c8-4c81-a0f4-0f98a738896e-metrics-certs\") pod \"network-metrics-daemon-gvd8v\" (UID: \"b20534ab-b8c8-4c81-a0f4-0f98a738896e\") " pod="openshift-multus/network-metrics-daemon-gvd8v"
Jul 06 20:24:00.375845 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:24:00.375780    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e42d3c32-18ba-48dc-afb5-90ef7f0e2752-original-pull-secret podName:e42d3c32-18ba-48dc-afb5-90ef7f0e2752 nodeName:}" failed. No retries permitted until 2026-07-06 20:24:02.375739432 +0000 UTC m=+6.091850084 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e42d3c32-18ba-48dc-afb5-90ef7f0e2752-original-pull-secret") pod "global-pull-secret-syncer-nzf7w" (UID: "e42d3c32-18ba-48dc-afb5-90ef7f0e2752") : object "kube-system"/"original-pull-secret" not registered
Jul 06 20:24:00.375845 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:24:00.375792    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 20:24:00.376001 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:24:00.375855    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b20534ab-b8c8-4c81-a0f4-0f98a738896e-metrics-certs podName:b20534ab-b8c8-4c81-a0f4-0f98a738896e nodeName:}" failed. No retries permitted until 2026-07-06 20:24:02.375841705 +0000 UTC m=+6.091952346 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b20534ab-b8c8-4c81-a0f4-0f98a738896e-metrics-certs") pod "network-metrics-daemon-gvd8v" (UID: "b20534ab-b8c8-4c81-a0f4-0f98a738896e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 20:24:00.476486 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:00.476428    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9wr4\" (UniqueName: \"kubernetes.io/projected/d2969391-caa9-4c3d-932c-3727d57b6ce8-kube-api-access-m9wr4\") pod \"network-check-target-tc5ls\" (UID: \"d2969391-caa9-4c3d-932c-3727d57b6ce8\") " pod="openshift-network-diagnostics/network-check-target-tc5ls"
Jul 06 20:24:00.476712 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:24:00.476680    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 06 20:24:00.476712 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:24:00.476710    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 06 20:24:00.476926 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:24:00.476725    2563 projected.go:196] Error preparing data for projected volume kube-api-access-m9wr4 for pod openshift-network-diagnostics/network-check-target-tc5ls: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 20:24:00.476926 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:24:00.476856    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d2969391-caa9-4c3d-932c-3727d57b6ce8-kube-api-access-m9wr4 podName:d2969391-caa9-4c3d-932c-3727d57b6ce8 nodeName:}" failed. No retries permitted until 2026-07-06 20:24:02.476775401 +0000 UTC m=+6.192886061 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-m9wr4" (UniqueName: "kubernetes.io/projected/d2969391-caa9-4c3d-932c-3727d57b6ce8-kube-api-access-m9wr4") pod "network-check-target-tc5ls" (UID: "d2969391-caa9-4c3d-932c-3727d57b6ce8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 20:24:00.558331 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:00.558238    2563 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 06 20:24:00.725331 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:00.725276    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-05 20:18:59 +0000 UTC" deadline="2028-01-28 16:04:05.402968776 +0000 UTC"
Jul 06 20:24:00.725331 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:00.725324    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13699h40m4.677651036s"
Jul 06 20:24:00.833581 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:00.833495    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gvd8v"
Jul 06 20:24:00.833760 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:24:00.833708    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gvd8v" podUID="b20534ab-b8c8-4c81-a0f4-0f98a738896e"
Jul 06 20:24:00.834289 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:00.834264    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nzf7w"
Jul 06 20:24:00.834459 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:24:00.834416    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nzf7w" podUID="e42d3c32-18ba-48dc-afb5-90ef7f0e2752"
Jul 06 20:24:00.834536 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:00.834518    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tc5ls"
Jul 06 20:24:00.834670 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:24:00.834642    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tc5ls" podUID="d2969391-caa9-4c3d-932c-3727d57b6ce8"
Jul 06 20:24:00.850461 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:00.850403    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ls5cs" event={"ID":"66288c77-335a-4c17-9ba7-bee3609820cb","Type":"ContainerStarted","Data":"261909b9019d66089020bff965387d00558341893f7bc40372f2f2e0900974b6"}
Jul 06 20:24:00.859698 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:00.859610    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-gl5px" event={"ID":"64a3d8f1-f091-4ba3-9f90-1b77cfea3c71","Type":"ContainerStarted","Data":"0d36dc6a26012e38bb55d6171c43254065bc8bbebb24d41c8897d8b859b13f0c"}
Jul 06 20:24:00.873779 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:00.873536    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-123.ec2.internal" event={"ID":"bc8b346ff26bd2da637db45021bfde3d","Type":"ContainerStarted","Data":"c94bf240019ac32627cf1dcf2756d706d4ab4ef76de3be2c8d1effa0f6f6fa21"}
Jul 06 20:24:00.889330 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:00.889278    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-48m7n" event={"ID":"7f8a1214-4b3e-4de6-96fe-8f9be6cca7f7","Type":"ContainerStarted","Data":"3c31ff108e4861a56ecb5ac94fdfa3192c6cb547a2fc6cf3eb221b244f45c9f6"}
Jul 06 20:24:00.891951 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:00.891547    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-123.ec2.internal" podStartSLOduration=2.8915281520000002 podStartE2EDuration="2.891528152s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:23:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:24:00.890419077 +0000 UTC m=+4.606529744" watchObservedRunningTime="2026-07-06 20:24:00.891528152 +0000 UTC m=+4.607638818"
Jul 06 20:24:00.902807 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:00.902769    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-2hwd9" event={"ID":"ed41c82c-bbcc-4385-8f89-f6ff340c2e36","Type":"ContainerStarted","Data":"c820514c2b7827cea236bb37d8eeb2f0a727a3777a5956f84ea9447d6a0a6069"}
Jul 06 20:24:00.916767 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:00.916675    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-2cqxl" event={"ID":"408f722a-5e7e-4bad-a86e-d461b5fed747","Type":"ContainerStarted","Data":"002a998b0c81f1e882286b40d3d4338fa4d75fa3f46b21929e2926b8597566df"}
Jul 06 20:24:00.950130 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:00.950091    2563 generic.go:356] "Generic (PLEG): container finished" podID="b7c519ecbb7e86b3494895d9cd797fde" containerID="16e6f351f3d2698d37bbf99ab30ea1fd3c49fe7a83e1934ce15f75e5ecca4680" exitCode=0
Jul 06 20:24:00.950452 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:00.950411    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-123.ec2.internal" event={"ID":"b7c519ecbb7e86b3494895d9cd797fde","Type":"ContainerDied","Data":"16e6f351f3d2698d37bbf99ab30ea1fd3c49fe7a83e1934ce15f75e5ecca4680"}
Jul 06 20:24:00.965128 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:00.965071    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5tk9w" event={"ID":"53799812-6492-4e52-94be-ce6e529cfc88","Type":"ContainerStarted","Data":"09edead3b454edc627311f59e1dd6f4682c6a20f4e5e8ae6b21221708c0807f9"}
Jul 06 20:24:00.969368 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:00.969292    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-7pkrs" event={"ID":"f48f979d-1c07-47ef-8144-488cad085d6e","Type":"ContainerStarted","Data":"f9cfad28a382001f74360872368989b6b42c7372d770b2287dac14c9b609721e"}
Jul 06 20:24:00.977392 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:00.974902    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-n9568" event={"ID":"1816531c-ac4a-4d49-96ac-234daa36ea91","Type":"ContainerStarted","Data":"ee400fdb2fc840844f8f10c91b940d16ae03ebeb5798bad8a4dac0e5b8c43c3d"}
Jul 06 20:24:00.977662 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:00.977492    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b7wcr" event={"ID":"048dd56c-9989-44f2-a1d5-c3c50db6938c","Type":"ContainerStarted","Data":"2c4fd307dcf304aff922db9d78cc12f1efdad1cf6bea0433774c92eb4879abde"}
Jul 06 20:24:02.004558 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:02.004517    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-123.ec2.internal" event={"ID":"b7c519ecbb7e86b3494895d9cd797fde","Type":"ContainerStarted","Data":"c7058e5455e23cac627876d23589e354429ce42cf386a501915203b23a8a9900"}
Jul 06 20:24:02.021212 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:02.021153    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-123.ec2.internal" podStartSLOduration=4.021127606 podStartE2EDuration="4.021127606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:23:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:24:02.020292134 +0000 UTC m=+5.736402799" watchObservedRunningTime="2026-07-06 20:24:02.021127606 +0000 UTC m=+5.737238271"
Jul 06 20:24:02.406667 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:02.406569    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e42d3c32-18ba-48dc-afb5-90ef7f0e2752-original-pull-secret\") pod \"global-pull-secret-syncer-nzf7w\" (UID: \"e42d3c32-18ba-48dc-afb5-90ef7f0e2752\") " pod="kube-system/global-pull-secret-syncer-nzf7w"
Jul 06 20:24:02.406667 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:02.406636    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b20534ab-b8c8-4c81-a0f4-0f98a738896e-metrics-certs\") pod \"network-metrics-daemon-gvd8v\" (UID: \"b20534ab-b8c8-4c81-a0f4-0f98a738896e\") " pod="openshift-multus/network-metrics-daemon-gvd8v"
Jul 06 20:24:02.406962 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:24:02.406844    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 20:24:02.406962 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:24:02.406914    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b20534ab-b8c8-4c81-a0f4-0f98a738896e-metrics-certs podName:b20534ab-b8c8-4c81-a0f4-0f98a738896e nodeName:}" failed. No retries permitted until 2026-07-06 20:24:06.406894181 +0000 UTC m=+10.123004844 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b20534ab-b8c8-4c81-a0f4-0f98a738896e-metrics-certs") pod "network-metrics-daemon-gvd8v" (UID: "b20534ab-b8c8-4c81-a0f4-0f98a738896e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 20:24:02.407373 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:24:02.407340    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 06 20:24:02.407458 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:24:02.407397    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e42d3c32-18ba-48dc-afb5-90ef7f0e2752-original-pull-secret podName:e42d3c32-18ba-48dc-afb5-90ef7f0e2752 nodeName:}" failed. No retries permitted until 2026-07-06 20:24:06.407380781 +0000 UTC m=+10.123491455 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e42d3c32-18ba-48dc-afb5-90ef7f0e2752-original-pull-secret") pod "global-pull-secret-syncer-nzf7w" (UID: "e42d3c32-18ba-48dc-afb5-90ef7f0e2752") : object "kube-system"/"original-pull-secret" not registered
Jul 06 20:24:02.507565 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:02.507522    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9wr4\" (UniqueName: \"kubernetes.io/projected/d2969391-caa9-4c3d-932c-3727d57b6ce8-kube-api-access-m9wr4\") pod \"network-check-target-tc5ls\" (UID: \"d2969391-caa9-4c3d-932c-3727d57b6ce8\") " pod="openshift-network-diagnostics/network-check-target-tc5ls"
Jul 06 20:24:02.507796 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:24:02.507732    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 06 20:24:02.507796 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:24:02.507752    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 06 20:24:02.507796 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:24:02.507764    2563 projected.go:196] Error preparing data for projected volume kube-api-access-m9wr4 for pod openshift-network-diagnostics/network-check-target-tc5ls: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 20:24:02.507970 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:24:02.507844    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d2969391-caa9-4c3d-932c-3727d57b6ce8-kube-api-access-m9wr4 podName:d2969391-caa9-4c3d-932c-3727d57b6ce8 nodeName:}" failed. No retries permitted until 2026-07-06 20:24:06.507823855 +0000 UTC m=+10.223934508 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-m9wr4" (UniqueName: "kubernetes.io/projected/d2969391-caa9-4c3d-932c-3727d57b6ce8-kube-api-access-m9wr4") pod "network-check-target-tc5ls" (UID: "d2969391-caa9-4c3d-932c-3727d57b6ce8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 20:24:02.831070 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:02.830981    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gvd8v"
Jul 06 20:24:02.831250 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:24:02.831137    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gvd8v" podUID="b20534ab-b8c8-4c81-a0f4-0f98a738896e"
Jul 06 20:24:02.831567 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:02.831544    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tc5ls"
Jul 06 20:24:02.831677 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:24:02.831656    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tc5ls" podUID="d2969391-caa9-4c3d-932c-3727d57b6ce8"
Jul 06 20:24:02.831737 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:02.831724    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nzf7w"
Jul 06 20:24:02.831842 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:24:02.831822    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nzf7w" podUID="e42d3c32-18ba-48dc-afb5-90ef7f0e2752"
Jul 06 20:24:04.833989 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:04.833627    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tc5ls"
Jul 06 20:24:04.833989 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:24:04.833750    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tc5ls" podUID="d2969391-caa9-4c3d-932c-3727d57b6ce8"
Jul 06 20:24:04.836786 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:04.836515    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nzf7w"
Jul 06 20:24:04.836786 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:24:04.836681    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nzf7w" podUID="e42d3c32-18ba-48dc-afb5-90ef7f0e2752"
Jul 06 20:24:04.836786 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:04.836767    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gvd8v"
Jul 06 20:24:04.837057 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:24:04.836989    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gvd8v" podUID="b20534ab-b8c8-4c81-a0f4-0f98a738896e"
Jul 06 20:24:06.440910 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:06.440869    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e42d3c32-18ba-48dc-afb5-90ef7f0e2752-original-pull-secret\") pod \"global-pull-secret-syncer-nzf7w\" (UID: \"e42d3c32-18ba-48dc-afb5-90ef7f0e2752\") " pod="kube-system/global-pull-secret-syncer-nzf7w"
Jul 06 20:24:06.440910 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:06.440933    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b20534ab-b8c8-4c81-a0f4-0f98a738896e-metrics-certs\") pod \"network-metrics-daemon-gvd8v\" (UID: \"b20534ab-b8c8-4c81-a0f4-0f98a738896e\") " pod="openshift-multus/network-metrics-daemon-gvd8v"
Jul 06 20:24:06.441837 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:24:06.441100    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 20:24:06.441837 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:24:06.441162    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b20534ab-b8c8-4c81-a0f4-0f98a738896e-metrics-certs podName:b20534ab-b8c8-4c81-a0f4-0f98a738896e nodeName:}" failed. No retries permitted until 2026-07-06 20:24:14.441143404 +0000 UTC m=+18.157254050 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b20534ab-b8c8-4c81-a0f4-0f98a738896e-metrics-certs") pod "network-metrics-daemon-gvd8v" (UID: "b20534ab-b8c8-4c81-a0f4-0f98a738896e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 20:24:06.441837 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:24:06.441240    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 06 20:24:06.441837 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:24:06.441276    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e42d3c32-18ba-48dc-afb5-90ef7f0e2752-original-pull-secret podName:e42d3c32-18ba-48dc-afb5-90ef7f0e2752 nodeName:}" failed. No retries permitted until 2026-07-06 20:24:14.441264084 +0000 UTC m=+18.157374727 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e42d3c32-18ba-48dc-afb5-90ef7f0e2752-original-pull-secret") pod "global-pull-secret-syncer-nzf7w" (UID: "e42d3c32-18ba-48dc-afb5-90ef7f0e2752") : object "kube-system"/"original-pull-secret" not registered
Jul 06 20:24:06.541971 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:06.541929    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9wr4\" (UniqueName: \"kubernetes.io/projected/d2969391-caa9-4c3d-932c-3727d57b6ce8-kube-api-access-m9wr4\") pod \"network-check-target-tc5ls\" (UID: \"d2969391-caa9-4c3d-932c-3727d57b6ce8\") " pod="openshift-network-diagnostics/network-check-target-tc5ls"
Jul 06 20:24:06.542163 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:24:06.542142    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 06 20:24:06.542228 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:24:06.542169    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 06 20:24:06.542228 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:24:06.542183    2563 projected.go:196] Error preparing data for projected volume kube-api-access-m9wr4 for pod openshift-network-diagnostics/network-check-target-tc5ls: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 20:24:06.542342 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:24:06.542251    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d2969391-caa9-4c3d-932c-3727d57b6ce8-kube-api-access-m9wr4 podName:d2969391-caa9-4c3d-932c-3727d57b6ce8 nodeName:}" failed. No retries permitted until 2026-07-06 20:24:14.54222897 +0000 UTC m=+18.258339668 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-m9wr4" (UniqueName: "kubernetes.io/projected/d2969391-caa9-4c3d-932c-3727d57b6ce8-kube-api-access-m9wr4") pod "network-check-target-tc5ls" (UID: "d2969391-caa9-4c3d-932c-3727d57b6ce8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 20:24:06.829129 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:06.829044    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gvd8v"
Jul 06 20:24:06.829310 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:24:06.829147    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gvd8v" podUID="b20534ab-b8c8-4c81-a0f4-0f98a738896e"
Jul 06 20:24:06.829589 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:06.829491    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tc5ls"
Jul 06 20:24:06.829589 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:24:06.829564    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tc5ls" podUID="d2969391-caa9-4c3d-932c-3727d57b6ce8"
Jul 06 20:24:06.829787 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:06.829708    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nzf7w"
Jul 06 20:24:06.829863 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:24:06.829822    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nzf7w" podUID="e42d3c32-18ba-48dc-afb5-90ef7f0e2752"
Jul 06 20:24:08.828668 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:08.828570    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gvd8v"
Jul 06 20:24:08.828668 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:08.828639    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tc5ls"
Jul 06 20:24:08.829191 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:24:08.828746    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gvd8v" podUID="b20534ab-b8c8-4c81-a0f4-0f98a738896e"
Jul 06 20:24:08.829191 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:08.828819    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nzf7w"
Jul 06 20:24:08.829191 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:24:08.828817    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tc5ls" podUID="d2969391-caa9-4c3d-932c-3727d57b6ce8"
Jul 06 20:24:08.829191 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:24:08.828925    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nzf7w" podUID="e42d3c32-18ba-48dc-afb5-90ef7f0e2752"
Jul 06 20:24:10.827783 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:10.827750    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gvd8v"
Jul 06 20:24:10.827783 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:10.827771    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nzf7w"
Jul 06 20:24:10.828282 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:10.827750    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tc5ls"
Jul 06 20:24:10.828282 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:24:10.827876    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gvd8v" podUID="b20534ab-b8c8-4c81-a0f4-0f98a738896e"
Jul 06 20:24:10.828282 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:24:10.827982    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tc5ls" podUID="d2969391-caa9-4c3d-932c-3727d57b6ce8"
Jul 06 20:24:10.828282 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:24:10.828056    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nzf7w" podUID="e42d3c32-18ba-48dc-afb5-90ef7f0e2752"
Jul 06 20:24:12.827986 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:12.827942    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nzf7w"
Jul 06 20:24:12.828457 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:12.827957    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tc5ls"
Jul 06 20:24:12.828457 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:24:12.828077    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nzf7w" podUID="e42d3c32-18ba-48dc-afb5-90ef7f0e2752"
Jul 06 20:24:12.828457 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:12.827957    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gvd8v"
Jul 06 20:24:12.828457 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:24:12.828149    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tc5ls" podUID="d2969391-caa9-4c3d-932c-3727d57b6ce8"
Jul 06 20:24:12.828457 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:24:12.828205    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gvd8v" podUID="b20534ab-b8c8-4c81-a0f4-0f98a738896e"
Jul 06 20:24:14.504654 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:14.504607    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b20534ab-b8c8-4c81-a0f4-0f98a738896e-metrics-certs\") pod \"network-metrics-daemon-gvd8v\" (UID: \"b20534ab-b8c8-4c81-a0f4-0f98a738896e\") " pod="openshift-multus/network-metrics-daemon-gvd8v"
Jul 06 20:24:14.505174 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:14.504701    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e42d3c32-18ba-48dc-afb5-90ef7f0e2752-original-pull-secret\") pod \"global-pull-secret-syncer-nzf7w\" (UID: \"e42d3c32-18ba-48dc-afb5-90ef7f0e2752\") " pod="kube-system/global-pull-secret-syncer-nzf7w"
Jul 06 20:24:14.505174 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:24:14.504812    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 20:24:14.505174 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:24:14.504849    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 06 20:24:14.505174 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:24:14.504887    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b20534ab-b8c8-4c81-a0f4-0f98a738896e-metrics-certs podName:b20534ab-b8c8-4c81-a0f4-0f98a738896e nodeName:}" failed. No retries permitted until 2026-07-06 20:24:30.504867942 +0000 UTC m=+34.220978584 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b20534ab-b8c8-4c81-a0f4-0f98a738896e-metrics-certs") pod "network-metrics-daemon-gvd8v" (UID: "b20534ab-b8c8-4c81-a0f4-0f98a738896e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 20:24:14.505174 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:24:14.504902    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e42d3c32-18ba-48dc-afb5-90ef7f0e2752-original-pull-secret podName:e42d3c32-18ba-48dc-afb5-90ef7f0e2752 nodeName:}" failed. No retries permitted until 2026-07-06 20:24:30.504895576 +0000 UTC m=+34.221006214 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e42d3c32-18ba-48dc-afb5-90ef7f0e2752-original-pull-secret") pod "global-pull-secret-syncer-nzf7w" (UID: "e42d3c32-18ba-48dc-afb5-90ef7f0e2752") : object "kube-system"/"original-pull-secret" not registered
Jul 06 20:24:14.605224 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:14.605180    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9wr4\" (UniqueName: \"kubernetes.io/projected/d2969391-caa9-4c3d-932c-3727d57b6ce8-kube-api-access-m9wr4\") pod \"network-check-target-tc5ls\" (UID: \"d2969391-caa9-4c3d-932c-3727d57b6ce8\") " pod="openshift-network-diagnostics/network-check-target-tc5ls"
Jul 06 20:24:14.605395 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:24:14.605370    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 06 20:24:14.605443 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:24:14.605404    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 06 20:24:14.605443 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:24:14.605416    2563 projected.go:196] Error preparing data for projected volume kube-api-access-m9wr4 for pod openshift-network-diagnostics/network-check-target-tc5ls: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 20:24:14.605521 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:24:14.605464    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d2969391-caa9-4c3d-932c-3727d57b6ce8-kube-api-access-m9wr4 podName:d2969391-caa9-4c3d-932c-3727d57b6ce8 nodeName:}" failed. No retries permitted until 2026-07-06 20:24:30.605449834 +0000 UTC m=+34.321560477 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-m9wr4" (UniqueName: "kubernetes.io/projected/d2969391-caa9-4c3d-932c-3727d57b6ce8-kube-api-access-m9wr4") pod "network-check-target-tc5ls" (UID: "d2969391-caa9-4c3d-932c-3727d57b6ce8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 20:24:14.828967 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:14.828836    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nzf7w"
Jul 06 20:24:14.828967 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:14.828841    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gvd8v"
Jul 06 20:24:14.829177 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:24:14.828974    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nzf7w" podUID="e42d3c32-18ba-48dc-afb5-90ef7f0e2752"
Jul 06 20:24:14.829177 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:24:14.829079    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gvd8v" podUID="b20534ab-b8c8-4c81-a0f4-0f98a738896e"
Jul 06 20:24:14.829177 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:14.828849    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tc5ls"
Jul 06 20:24:14.829319 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:24:14.829233    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tc5ls" podUID="d2969391-caa9-4c3d-932c-3727d57b6ce8"
Jul 06 20:24:16.828723 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:16.828689    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nzf7w"
Jul 06 20:24:16.829198 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:16.828801    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tc5ls"
Jul 06 20:24:16.829198 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:16.828837    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gvd8v"
Jul 06 20:24:16.829198 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:24:16.828847    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nzf7w" podUID="e42d3c32-18ba-48dc-afb5-90ef7f0e2752"
Jul 06 20:24:16.829198 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:24:16.828927    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gvd8v" podUID="b20534ab-b8c8-4c81-a0f4-0f98a738896e"
Jul 06 20:24:16.829198 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:24:16.829004    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tc5ls" podUID="d2969391-caa9-4c3d-932c-3727d57b6ce8"
Jul 06 20:24:18.828084 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:18.828053    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tc5ls"
Jul 06 20:24:18.828530 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:18.828050    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gvd8v"
Jul 06 20:24:18.828530 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:18.828177    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nzf7w"
Jul 06 20:24:18.828530 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:24:18.828173    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tc5ls" podUID="d2969391-caa9-4c3d-932c-3727d57b6ce8"
Jul 06 20:24:18.828530 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:24:18.828280    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gvd8v" podUID="b20534ab-b8c8-4c81-a0f4-0f98a738896e"
Jul 06 20:24:18.828530 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:24:18.828366    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nzf7w" podUID="e42d3c32-18ba-48dc-afb5-90ef7f0e2752"
Jul 06 20:24:20.045429 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:20.045168    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-2hwd9" event={"ID":"ed41c82c-bbcc-4385-8f89-f6ff340c2e36","Type":"ContainerStarted","Data":"3d5dbae0f7f5aec08648ec4027992db08e9fc98e74a88829e5aa517612392973"}
Jul 06 20:24:20.047278 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:20.047231    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-2cqxl" event={"ID":"408f722a-5e7e-4bad-a86e-d461b5fed747","Type":"ContainerStarted","Data":"1d68abd14a8089cb7e436e7ea5cd96528f91b3c8f1b4251618fecdf1a59a5e99"}
Jul 06 20:24:20.051329 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:20.051293    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5tk9w" event={"ID":"53799812-6492-4e52-94be-ce6e529cfc88","Type":"ContainerStarted","Data":"6b2790e9f7514e6505d2307b2bb2139ddbd771d10621ba5cd8b48ec4975bb36f"}
Jul 06 20:24:20.051424 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:20.051371    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5tk9w" event={"ID":"53799812-6492-4e52-94be-ce6e529cfc88","Type":"ContainerStarted","Data":"4b9682dcccb90de94a45fd5103be6c5271cdecf0c290b7fb6b6a356c62733df4"}
Jul 06 20:24:20.051479 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:20.051422    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5tk9w" event={"ID":"53799812-6492-4e52-94be-ce6e529cfc88","Type":"ContainerStarted","Data":"12976c4f9ebf624cb1fa07afc1d9a5f45625bfd205cb26ef99542a77428c40fe"}
Jul 06 20:24:20.051479 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:20.051471    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5tk9w" event={"ID":"53799812-6492-4e52-94be-ce6e529cfc88","Type":"ContainerStarted","Data":"95853dc441005d0454933004ffc8b8e7fb0b3a66aeeb35918f76a1c5da476c17"}
Jul 06 20:24:20.051591 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:20.051517    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5tk9w" event={"ID":"53799812-6492-4e52-94be-ce6e529cfc88","Type":"ContainerStarted","Data":"d7e7f5f4fc0d3ac16b760ba645027834c1af539f079f17cb72ce64f688b8946a"}
Jul 06 20:24:20.051591 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:20.051552    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5tk9w" event={"ID":"53799812-6492-4e52-94be-ce6e529cfc88","Type":"ContainerStarted","Data":"1a536dfb21b7b53a19f12dd0390366224d947fc1e0d9771bc8483bcf8fbc2a4f"}
Jul 06 20:24:20.053005 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:20.052935    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-7pkrs" event={"ID":"f48f979d-1c07-47ef-8144-488cad085d6e","Type":"ContainerStarted","Data":"3805eab355a70cb8bd27a3d50c41e92eb1b7b8e2ed1071a12d4d92a4738c7883"}
Jul 06 20:24:20.054401 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:20.054374    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-n9568" event={"ID":"1816531c-ac4a-4d49-96ac-234daa36ea91","Type":"ContainerStarted","Data":"8eee432a9cce88b589065ee968c745cee96275d5acb5bea69a40722d1a21a47d"}
Jul 06 20:24:20.056065 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:20.056043    2563 generic.go:356] "Generic (PLEG): container finished" podID="048dd56c-9989-44f2-a1d5-c3c50db6938c" containerID="af95c5a78afb54eacaa8d6d0114d7e1a18c4571ff439940af344c51c5bde46a3" exitCode=0
Jul 06 20:24:20.056162 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:20.056119    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b7wcr" event={"ID":"048dd56c-9989-44f2-a1d5-c3c50db6938c","Type":"ContainerDied","Data":"af95c5a78afb54eacaa8d6d0114d7e1a18c4571ff439940af344c51c5bde46a3"}
Jul 06 20:24:20.057597 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:20.057574    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ls5cs" event={"ID":"66288c77-335a-4c17-9ba7-bee3609820cb","Type":"ContainerStarted","Data":"6eb4d7dca944472b68c7b8010d3222b56f24e9cb481d51d4b8e0a036507ab848"}
Jul 06 20:24:20.059264 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:20.059239    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-gl5px" event={"ID":"64a3d8f1-f091-4ba3-9f90-1b77cfea3c71","Type":"ContainerStarted","Data":"4a1972456b3201b42eda95f85a886eb0e520fda2e76e78390f47177dda4b2241"}
Jul 06 20:24:20.061809 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:20.061777    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-2hwd9" podStartSLOduration=3.810173267 podStartE2EDuration="23.061766368s" totalImagesPullingTime="19.251593101s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:23:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:24:20.06143831 +0000 UTC m=+23.777548971" watchObservedRunningTime="2026-07-06 20:24:20.061766368 +0000 UTC m=+23.777877030"
Jul 06 20:24:20.091549 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:20.091490    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-gl5px" podStartSLOduration=3.775553143 podStartE2EDuration="23.091466485s" totalImagesPullingTime="19.315913342s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:23:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:24:20.091199998 +0000 UTC m=+23.807310677" watchObservedRunningTime="2026-07-06 20:24:20.091466485 +0000 UTC m=+23.807577171"
Jul 06 20:24:20.167516 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:20.167464    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-n9568" podStartSLOduration=3.915570964 podStartE2EDuration="23.167449168s" totalImagesPullingTime="19.251878204s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:23:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:24:20.167239942 +0000 UTC m=+23.883350604" watchObservedRunningTime="2026-07-06 20:24:20.167449168 +0000 UTC m=+23.883559832"
Jul 06 20:24:20.213883 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:20.213831    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-7pkrs" podStartSLOduration=3.953132121 podStartE2EDuration="23.213813147s" totalImagesPullingTime="19.260681026s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:23:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:24:20.211800244 +0000 UTC m=+23.927910906" watchObservedRunningTime="2026-07-06 20:24:20.213813147 +0000 UTC m=+23.929923811"
Jul 06 20:24:20.249636 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:20.249589    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-2cqxl" podStartSLOduration=3.995742016 podStartE2EDuration="23.249573891s" totalImagesPullingTime="19.253831875s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:23:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:24:20.249340978 +0000 UTC m=+23.965451640" watchObservedRunningTime="2026-07-06 20:24:20.249573891 +0000 UTC m=+23.965684552"
Jul 06 20:24:20.384587 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:20.384540    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:24:20.675660 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:20.675634    2563 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 06 20:24:20.713687 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:20.713537    2563 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-06T20:24:20.675656536Z","UUID":"6bde89a6-63af-4715-8966-5ed6492c5712","Handler":null,"Name":"","Endpoint":""}
Jul 06 20:24:20.715079 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:20.715059    2563 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 06 20:24:20.715174 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:20.715099    2563 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 06 20:24:20.828849 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:20.828810    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tc5ls"
Jul 06 20:24:20.829023 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:20.828931    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nzf7w"
Jul 06 20:24:20.829023 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:24:20.828937    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tc5ls" podUID="d2969391-caa9-4c3d-932c-3727d57b6ce8"
Jul 06 20:24:20.829023 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:20.828965    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gvd8v"
Jul 06 20:24:20.829173 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:24:20.829056    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nzf7w" podUID="e42d3c32-18ba-48dc-afb5-90ef7f0e2752"
Jul 06 20:24:20.829173 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:24:20.829115    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gvd8v" podUID="b20534ab-b8c8-4c81-a0f4-0f98a738896e"
Jul 06 20:24:21.063946 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:21.063860    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-48m7n" event={"ID":"7f8a1214-4b3e-4de6-96fe-8f9be6cca7f7","Type":"ContainerStarted","Data":"8a1864b4ed6b86f21328ba3d32181ec66a6bd3fb523723003aeb37af59d74e6d"}
Jul 06 20:24:21.065927 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:21.065900    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ls5cs" event={"ID":"66288c77-335a-4c17-9ba7-bee3609820cb","Type":"ContainerStarted","Data":"62898545464c6debebe6a9b604492853bfe4581fbcf8ab9132c0a829fd16e027"}
Jul 06 20:24:21.621468 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:21.621437    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-2cqxl"
Jul 06 20:24:21.621957 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:21.621937    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-2cqxl"
Jul 06 20:24:21.642372 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:21.642332    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-48m7n" podStartSLOduration=4.38358638 podStartE2EDuration="23.642316638s" totalImagesPullingTime="19.258730258s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:23:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:24:21.082753395 +0000 UTC m=+24.798864056" watchObservedRunningTime="2026-07-06 20:24:21.642316638 +0000 UTC m=+25.358427298"
Jul 06 20:24:21.868525 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:21.868496    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:24:22.070528 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:22.070454    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ls5cs" event={"ID":"66288c77-335a-4c17-9ba7-bee3609820cb","Type":"ContainerStarted","Data":"07567124f06dfc6862ef4ba054ff6543aff1fe6998a6c9b9f3404619881a72ca"}
Jul 06 20:24:22.071105 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:22.071078    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-2cqxl"
Jul 06 20:24:22.071653 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:22.071628    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-2cqxl"
Jul 06 20:24:22.122254 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:22.122208    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ls5cs" podStartSLOduration=2.817053707 podStartE2EDuration="24.122193768s" totalImagesPullingTime="21.305140061s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:23:58 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:24:22.103396047 +0000 UTC m=+25.819506709" watchObservedRunningTime="2026-07-06 20:24:22.122193768 +0000 UTC m=+25.838304446"
Jul 06 20:24:22.178061 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:22.178024    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:24:22.391042 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:22.390959    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:24:22.490901 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:22.490869    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:24:22.771880 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:22.771781    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:24:22.827990 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:22.827965    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gvd8v"
Jul 06 20:24:22.828177 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:22.827965    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tc5ls"
Jul 06 20:24:22.828177 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:22.828078    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nzf7w"
Jul 06 20:24:22.828177 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:24:22.828075    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gvd8v" podUID="b20534ab-b8c8-4c81-a0f4-0f98a738896e"
Jul 06 20:24:22.828177 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:24:22.828154    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tc5ls" podUID="d2969391-caa9-4c3d-932c-3727d57b6ce8"
Jul 06 20:24:22.828381 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:24:22.828237    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nzf7w" podUID="e42d3c32-18ba-48dc-afb5-90ef7f0e2752"
Jul 06 20:24:22.872512 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:22.872480    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:24:23.076925 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:23.076849    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5tk9w" event={"ID":"53799812-6492-4e52-94be-ce6e529cfc88","Type":"ContainerStarted","Data":"1049231c6566cbd718e303533f2f90c0e9698b79f77bed56f21f3ca42092e6f3"}
Jul 06 20:24:24.828819 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:24.828600    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nzf7w"
Jul 06 20:24:24.829252 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:24.828600    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gvd8v"
Jul 06 20:24:24.829252 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:24:24.828919    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nzf7w" podUID="e42d3c32-18ba-48dc-afb5-90ef7f0e2752"
Jul 06 20:24:24.829252 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:24:24.829012    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gvd8v" podUID="b20534ab-b8c8-4c81-a0f4-0f98a738896e"
Jul 06 20:24:24.829252 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:24.828601    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tc5ls"
Jul 06 20:24:24.829252 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:24:24.829093    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tc5ls" podUID="d2969391-caa9-4c3d-932c-3727d57b6ce8"
Jul 06 20:24:25.498124 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:25.498092    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:24:26.086340 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:26.086035    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5tk9w" event={"ID":"53799812-6492-4e52-94be-ce6e529cfc88","Type":"ContainerStarted","Data":"e5716ed8affca4f3afcc4797bee77709fdfd8645bb65635ae307a53d9c8a6f09"}
Jul 06 20:24:26.086340 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:26.086308    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5tk9w"
Jul 06 20:24:26.088030 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:26.088002    2563 generic.go:356] "Generic (PLEG): container finished" podID="048dd56c-9989-44f2-a1d5-c3c50db6938c" containerID="3ab92479d41f1131e775327009bc4e3f2e81b38279c02352e43743d10e962785" exitCode=0
Jul 06 20:24:26.088143 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:26.088057    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b7wcr" event={"ID":"048dd56c-9989-44f2-a1d5-c3c50db6938c","Type":"ContainerDied","Data":"3ab92479d41f1131e775327009bc4e3f2e81b38279c02352e43743d10e962785"}
Jul 06 20:24:26.101861 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:26.101832    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5tk9w"
Jul 06 20:24:26.115138 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:26.115097    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-5tk9w" podStartSLOduration=9.789373026 podStartE2EDuration="29.115084254s" totalImagesPullingTime="19.325711228s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:23:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:24:26.114745909 +0000 UTC m=+29.830856567" watchObservedRunningTime="2026-07-06 20:24:26.115084254 +0000 UTC m=+29.831194914"
Jul 06 20:24:26.842431 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:26.842248    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gvd8v"
Jul 06 20:24:26.842431 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:24:26.842367    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gvd8v" podUID="b20534ab-b8c8-4c81-a0f4-0f98a738896e"
Jul 06 20:24:26.842869 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:26.842823    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tc5ls"
Jul 06 20:24:26.842966 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:24:26.842941    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tc5ls" podUID="d2969391-caa9-4c3d-932c-3727d57b6ce8"
Jul 06 20:24:26.843031 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:26.843004    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nzf7w"
Jul 06 20:24:26.843102 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:24:26.843075    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nzf7w" podUID="e42d3c32-18ba-48dc-afb5-90ef7f0e2752"
Jul 06 20:24:27.090272 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:27.090242    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5tk9w"
Jul 06 20:24:27.090638 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:27.090305    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5tk9w"
Jul 06 20:24:27.105922 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:27.105898    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5tk9w"
Jul 06 20:24:27.784066 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:27.784033    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-gvd8v"]
Jul 06 20:24:27.784221 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:27.784163    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gvd8v"
Jul 06 20:24:27.784304 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:24:27.784286    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gvd8v" podUID="b20534ab-b8c8-4c81-a0f4-0f98a738896e"
Jul 06 20:24:27.786878 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:27.786846    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-nzf7w"]
Jul 06 20:24:27.787039 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:27.786945    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nzf7w"
Jul 06 20:24:27.787095 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:24:27.787047    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nzf7w" podUID="e42d3c32-18ba-48dc-afb5-90ef7f0e2752"
Jul 06 20:24:27.787477 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:27.787452    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-tc5ls"]
Jul 06 20:24:27.787594 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:27.787538    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tc5ls"
Jul 06 20:24:27.787660 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:24:27.787625    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tc5ls" podUID="d2969391-caa9-4c3d-932c-3727d57b6ce8"
Jul 06 20:24:28.093714 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:28.093685    2563 generic.go:356] "Generic (PLEG): container finished" podID="048dd56c-9989-44f2-a1d5-c3c50db6938c" containerID="5fb889a131c7c6abb16913675e3c7ab6b6d0bd9a9a4bac7ed7f0eb668f7d7c89" exitCode=0
Jul 06 20:24:28.094083 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:28.093771    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b7wcr" event={"ID":"048dd56c-9989-44f2-a1d5-c3c50db6938c","Type":"ContainerDied","Data":"5fb889a131c7c6abb16913675e3c7ab6b6d0bd9a9a4bac7ed7f0eb668f7d7c89"}
Jul 06 20:24:29.828689 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:29.828443    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tc5ls"
Jul 06 20:24:29.829151 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:29.828453    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gvd8v"
Jul 06 20:24:29.829151 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:24:29.828723    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tc5ls" podUID="d2969391-caa9-4c3d-932c-3727d57b6ce8"
Jul 06 20:24:29.829151 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:29.828497    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nzf7w"
Jul 06 20:24:29.829151 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:24:29.828801    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gvd8v" podUID="b20534ab-b8c8-4c81-a0f4-0f98a738896e"
Jul 06 20:24:29.829151 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:24:29.828845    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nzf7w" podUID="e42d3c32-18ba-48dc-afb5-90ef7f0e2752"
Jul 06 20:24:30.100927 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:30.100849    2563 generic.go:356] "Generic (PLEG): container finished" podID="048dd56c-9989-44f2-a1d5-c3c50db6938c" containerID="23241c01671343eec2a25fb3c79e41127d8b7ea596bf67c0cd3eaaa7276e2d97" exitCode=0
Jul 06 20:24:30.100927 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:30.100896    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b7wcr" event={"ID":"048dd56c-9989-44f2-a1d5-c3c50db6938c","Type":"ContainerDied","Data":"23241c01671343eec2a25fb3c79e41127d8b7ea596bf67c0cd3eaaa7276e2d97"}
Jul 06 20:24:30.528007 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:30.527917    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b20534ab-b8c8-4c81-a0f4-0f98a738896e-metrics-certs\") pod \"network-metrics-daemon-gvd8v\" (UID: \"b20534ab-b8c8-4c81-a0f4-0f98a738896e\") " pod="openshift-multus/network-metrics-daemon-gvd8v"
Jul 06 20:24:30.528007 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:30.527994    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e42d3c32-18ba-48dc-afb5-90ef7f0e2752-original-pull-secret\") pod \"global-pull-secret-syncer-nzf7w\" (UID: \"e42d3c32-18ba-48dc-afb5-90ef7f0e2752\") " pod="kube-system/global-pull-secret-syncer-nzf7w"
Jul 06 20:24:30.528199 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:24:30.528081    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 20:24:30.528199 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:24:30.528150    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b20534ab-b8c8-4c81-a0f4-0f98a738896e-metrics-certs podName:b20534ab-b8c8-4c81-a0f4-0f98a738896e nodeName:}" failed. No retries permitted until 2026-07-06 20:25:02.528134362 +0000 UTC m=+66.244245005 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b20534ab-b8c8-4c81-a0f4-0f98a738896e-metrics-certs") pod "network-metrics-daemon-gvd8v" (UID: "b20534ab-b8c8-4c81-a0f4-0f98a738896e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 20:24:30.528199 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:24:30.528085    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 06 20:24:30.528297 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:24:30.528232    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e42d3c32-18ba-48dc-afb5-90ef7f0e2752-original-pull-secret podName:e42d3c32-18ba-48dc-afb5-90ef7f0e2752 nodeName:}" failed. No retries permitted until 2026-07-06 20:25:02.528219693 +0000 UTC m=+66.244330331 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e42d3c32-18ba-48dc-afb5-90ef7f0e2752-original-pull-secret") pod "global-pull-secret-syncer-nzf7w" (UID: "e42d3c32-18ba-48dc-afb5-90ef7f0e2752") : object "kube-system"/"original-pull-secret" not registered
Jul 06 20:24:30.628646 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:30.628613    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9wr4\" (UniqueName: \"kubernetes.io/projected/d2969391-caa9-4c3d-932c-3727d57b6ce8-kube-api-access-m9wr4\") pod \"network-check-target-tc5ls\" (UID: \"d2969391-caa9-4c3d-932c-3727d57b6ce8\") " pod="openshift-network-diagnostics/network-check-target-tc5ls"
Jul 06 20:24:30.628869 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:24:30.628787    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 06 20:24:30.628869 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:24:30.628811    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 06 20:24:30.628869 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:24:30.628824    2563 projected.go:196] Error preparing data for projected volume kube-api-access-m9wr4 for pod openshift-network-diagnostics/network-check-target-tc5ls: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 20:24:30.629028 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:24:30.628888    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d2969391-caa9-4c3d-932c-3727d57b6ce8-kube-api-access-m9wr4 podName:d2969391-caa9-4c3d-932c-3727d57b6ce8 nodeName:}" failed. No retries permitted until 2026-07-06 20:25:02.628869587 +0000 UTC m=+66.344980228 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-m9wr4" (UniqueName: "kubernetes.io/projected/d2969391-caa9-4c3d-932c-3727d57b6ce8-kube-api-access-m9wr4") pod "network-check-target-tc5ls" (UID: "d2969391-caa9-4c3d-932c-3727d57b6ce8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 20:24:31.828027 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:31.827976    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gvd8v"
Jul 06 20:24:31.828444 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:31.827977    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tc5ls"
Jul 06 20:24:31.828444 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:24:31.828117    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gvd8v" podUID="b20534ab-b8c8-4c81-a0f4-0f98a738896e"
Jul 06 20:24:31.828444 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:31.827988    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nzf7w"
Jul 06 20:24:31.828444 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:24:31.828161    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tc5ls" podUID="d2969391-caa9-4c3d-932c-3727d57b6ce8"
Jul 06 20:24:31.828444 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:24:31.828247    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nzf7w" podUID="e42d3c32-18ba-48dc-afb5-90ef7f0e2752"
Jul 06 20:24:32.976665 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:32.976602    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:24:33.040173 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:33.040136    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:24:33.079021 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:33.078943    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-123.ec2.internal" event="NodeReady"
Jul 06 20:24:33.079156 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:33.079063    2563 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 06 20:24:33.131258 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:33.131225    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-gqvcm"]
Jul 06 20:24:33.133537 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:33.133511    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-fn7x6"]
Jul 06 20:24:33.133690 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:33.133656    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-gqvcm"
Jul 06 20:24:33.138595 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:33.135753    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-fn7x6"
Jul 06 20:24:33.139516 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:33.139216    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 06 20:24:33.139516 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:33.139240    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 06 20:24:33.139516 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:33.139426    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-mqhkz\""
Jul 06 20:24:33.141082 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:33.141058    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 06 20:24:33.141216 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:33.141100    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-gmpj7\""
Jul 06 20:24:33.141327 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:33.141310    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 06 20:24:33.141584 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:33.141565    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 06 20:24:33.143546 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:33.143360    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-fn7x6"]
Jul 06 20:24:33.146633 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:33.146614    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-gqvcm"]
Jul 06 20:24:33.253389 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:33.253355    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f2c17640-14e9-4b68-93a4-9183cbc2ccd0-config-volume\") pod \"dns-default-gqvcm\" (UID: \"f2c17640-14e9-4b68-93a4-9183cbc2ccd0\") " pod="openshift-dns/dns-default-gqvcm"
Jul 06 20:24:33.253389 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:33.253394    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f2c17640-14e9-4b68-93a4-9183cbc2ccd0-metrics-tls\") pod \"dns-default-gqvcm\" (UID: \"f2c17640-14e9-4b68-93a4-9183cbc2ccd0\") " pod="openshift-dns/dns-default-gqvcm"
Jul 06 20:24:33.253607 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:33.253436    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7w6n7\" (UniqueName: \"kubernetes.io/projected/f2c17640-14e9-4b68-93a4-9183cbc2ccd0-kube-api-access-7w6n7\") pod \"dns-default-gqvcm\" (UID: \"f2c17640-14e9-4b68-93a4-9183cbc2ccd0\") " pod="openshift-dns/dns-default-gqvcm"
Jul 06 20:24:33.253607 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:33.253473    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2c17640-14e9-4b68-93a4-9183cbc2ccd0-tmp-dir\") pod \"dns-default-gqvcm\" (UID: \"f2c17640-14e9-4b68-93a4-9183cbc2ccd0\") " pod="openshift-dns/dns-default-gqvcm"
Jul 06 20:24:33.253607 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:33.253493    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3cce0a2d-3fc6-4667-b38a-a0e800fbfe0b-cert\") pod \"ingress-canary-fn7x6\" (UID: \"3cce0a2d-3fc6-4667-b38a-a0e800fbfe0b\") " pod="openshift-ingress-canary/ingress-canary-fn7x6"
Jul 06 20:24:33.253607 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:33.253554    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sldc\" (UniqueName: \"kubernetes.io/projected/3cce0a2d-3fc6-4667-b38a-a0e800fbfe0b-kube-api-access-6sldc\") pod \"ingress-canary-fn7x6\" (UID: \"3cce0a2d-3fc6-4667-b38a-a0e800fbfe0b\") " pod="openshift-ingress-canary/ingress-canary-fn7x6"
Jul 06 20:24:33.269740 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:33.269699    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:24:33.327660 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:33.327618    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:24:33.354598 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:33.354519    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f2c17640-14e9-4b68-93a4-9183cbc2ccd0-config-volume\") pod \"dns-default-gqvcm\" (UID: \"f2c17640-14e9-4b68-93a4-9183cbc2ccd0\") " pod="openshift-dns/dns-default-gqvcm"
Jul 06 20:24:33.354598 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:33.354562    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f2c17640-14e9-4b68-93a4-9183cbc2ccd0-metrics-tls\") pod \"dns-default-gqvcm\" (UID: \"f2c17640-14e9-4b68-93a4-9183cbc2ccd0\") " pod="openshift-dns/dns-default-gqvcm"
Jul 06 20:24:33.354851 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:33.354602    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7w6n7\" (UniqueName: \"kubernetes.io/projected/f2c17640-14e9-4b68-93a4-9183cbc2ccd0-kube-api-access-7w6n7\") pod \"dns-default-gqvcm\" (UID: \"f2c17640-14e9-4b68-93a4-9183cbc2ccd0\") " pod="openshift-dns/dns-default-gqvcm"
Jul 06 20:24:33.354851 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:33.354644    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2c17640-14e9-4b68-93a4-9183cbc2ccd0-tmp-dir\") pod \"dns-default-gqvcm\" (UID: \"f2c17640-14e9-4b68-93a4-9183cbc2ccd0\") " pod="openshift-dns/dns-default-gqvcm"
Jul 06 20:24:33.354851 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:24:33.354772    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 06 20:24:33.354851 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:24:33.354844    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2c17640-14e9-4b68-93a4-9183cbc2ccd0-metrics-tls podName:f2c17640-14e9-4b68-93a4-9183cbc2ccd0 nodeName:}" failed. No retries permitted until 2026-07-06 20:24:33.85482265 +0000 UTC m=+37.570933305 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f2c17640-14e9-4b68-93a4-9183cbc2ccd0-metrics-tls") pod "dns-default-gqvcm" (UID: "f2c17640-14e9-4b68-93a4-9183cbc2ccd0") : secret "dns-default-metrics-tls" not found
Jul 06 20:24:33.355085 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:33.354946    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3cce0a2d-3fc6-4667-b38a-a0e800fbfe0b-cert\") pod \"ingress-canary-fn7x6\" (UID: \"3cce0a2d-3fc6-4667-b38a-a0e800fbfe0b\") " pod="openshift-ingress-canary/ingress-canary-fn7x6"
Jul 06 20:24:33.355085 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:33.354980    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2c17640-14e9-4b68-93a4-9183cbc2ccd0-tmp-dir\") pod \"dns-default-gqvcm\" (UID: \"f2c17640-14e9-4b68-93a4-9183cbc2ccd0\") " pod="openshift-dns/dns-default-gqvcm"
Jul 06 20:24:33.355085 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:33.354985    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sldc\" (UniqueName: \"kubernetes.io/projected/3cce0a2d-3fc6-4667-b38a-a0e800fbfe0b-kube-api-access-6sldc\") pod \"ingress-canary-fn7x6\" (UID: \"3cce0a2d-3fc6-4667-b38a-a0e800fbfe0b\") " pod="openshift-ingress-canary/ingress-canary-fn7x6"
Jul 06 20:24:33.355236 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:33.355198    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f2c17640-14e9-4b68-93a4-9183cbc2ccd0-config-volume\") pod \"dns-default-gqvcm\" (UID: \"f2c17640-14e9-4b68-93a4-9183cbc2ccd0\") " pod="openshift-dns/dns-default-gqvcm"
Jul 06 20:24:33.359794 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:33.359764    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3cce0a2d-3fc6-4667-b38a-a0e800fbfe0b-cert\") pod \"ingress-canary-fn7x6\" (UID: \"3cce0a2d-3fc6-4667-b38a-a0e800fbfe0b\") " pod="openshift-ingress-canary/ingress-canary-fn7x6"
Jul 06 20:24:33.364604 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:33.364563    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7w6n7\" (UniqueName: \"kubernetes.io/projected/f2c17640-14e9-4b68-93a4-9183cbc2ccd0-kube-api-access-7w6n7\") pod \"dns-default-gqvcm\" (UID: \"f2c17640-14e9-4b68-93a4-9183cbc2ccd0\") " pod="openshift-dns/dns-default-gqvcm"
Jul 06 20:24:33.364744 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:33.364578    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sldc\" (UniqueName: \"kubernetes.io/projected/3cce0a2d-3fc6-4667-b38a-a0e800fbfe0b-kube-api-access-6sldc\") pod \"ingress-canary-fn7x6\" (UID: \"3cce0a2d-3fc6-4667-b38a-a0e800fbfe0b\") " pod="openshift-ingress-canary/ingress-canary-fn7x6"
Jul 06 20:24:33.453712 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:33.453668    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-fn7x6"
Jul 06 20:24:33.603748 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:33.603703    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-fn7x6"]
Jul 06 20:24:33.607928 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:24:33.607902    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cce0a2d_3fc6_4667_b38a_a0e800fbfe0b.slice/crio-b21143b5fd43a113b0843b58c825eb6ec56fc6602daf3bbd7eacabe1f501b56c WatchSource:0}: Error finding container b21143b5fd43a113b0843b58c825eb6ec56fc6602daf3bbd7eacabe1f501b56c: Status 404 returned error can't find the container with id b21143b5fd43a113b0843b58c825eb6ec56fc6602daf3bbd7eacabe1f501b56c
Jul 06 20:24:33.828210 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:33.828001    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nzf7w"
Jul 06 20:24:33.828371 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:33.828001    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tc5ls"
Jul 06 20:24:33.828371 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:33.828001    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gvd8v"
Jul 06 20:24:33.830963 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:33.830938    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 06 20:24:33.831093 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:33.831055    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-hhd9j\""
Jul 06 20:24:33.831816 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:33.831792    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-z7zd8\""
Jul 06 20:24:33.831917 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:33.831792    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 06 20:24:33.831917 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:33.831789    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 06 20:24:33.831917 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:33.831800    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 06 20:24:33.859218 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:33.859160    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f2c17640-14e9-4b68-93a4-9183cbc2ccd0-metrics-tls\") pod \"dns-default-gqvcm\" (UID: \"f2c17640-14e9-4b68-93a4-9183cbc2ccd0\") " pod="openshift-dns/dns-default-gqvcm"
Jul 06 20:24:33.859321 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:24:33.859295    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 06 20:24:33.859367 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:24:33.859350    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2c17640-14e9-4b68-93a4-9183cbc2ccd0-metrics-tls podName:f2c17640-14e9-4b68-93a4-9183cbc2ccd0 nodeName:}" failed. No retries permitted until 2026-07-06 20:24:34.859335075 +0000 UTC m=+38.575445717 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f2c17640-14e9-4b68-93a4-9183cbc2ccd0-metrics-tls") pod "dns-default-gqvcm" (UID: "f2c17640-14e9-4b68-93a4-9183cbc2ccd0") : secret "dns-default-metrics-tls" not found
Jul 06 20:24:34.112464 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:34.112368    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-fn7x6" event={"ID":"3cce0a2d-3fc6-4667-b38a-a0e800fbfe0b","Type":"ContainerStarted","Data":"b21143b5fd43a113b0843b58c825eb6ec56fc6602daf3bbd7eacabe1f501b56c"}
Jul 06 20:24:34.868204 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:34.868156    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f2c17640-14e9-4b68-93a4-9183cbc2ccd0-metrics-tls\") pod \"dns-default-gqvcm\" (UID: \"f2c17640-14e9-4b68-93a4-9183cbc2ccd0\") " pod="openshift-dns/dns-default-gqvcm"
Jul 06 20:24:34.868400 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:24:34.868307    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 06 20:24:34.868400 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:24:34.868372    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2c17640-14e9-4b68-93a4-9183cbc2ccd0-metrics-tls podName:f2c17640-14e9-4b68-93a4-9183cbc2ccd0 nodeName:}" failed. No retries permitted until 2026-07-06 20:24:36.868353085 +0000 UTC m=+40.584463746 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f2c17640-14e9-4b68-93a4-9183cbc2ccd0-metrics-tls") pod "dns-default-gqvcm" (UID: "f2c17640-14e9-4b68-93a4-9183cbc2ccd0") : secret "dns-default-metrics-tls" not found
Jul 06 20:24:36.532960 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:36.532926    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:24:36.632718 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:36.632688    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:24:36.883331 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:36.883306    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f2c17640-14e9-4b68-93a4-9183cbc2ccd0-metrics-tls\") pod \"dns-default-gqvcm\" (UID: \"f2c17640-14e9-4b68-93a4-9183cbc2ccd0\") " pod="openshift-dns/dns-default-gqvcm"
Jul 06 20:24:36.883488 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:24:36.883468    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 06 20:24:36.883544 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:24:36.883531    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2c17640-14e9-4b68-93a4-9183cbc2ccd0-metrics-tls podName:f2c17640-14e9-4b68-93a4-9183cbc2ccd0 nodeName:}" failed. No retries permitted until 2026-07-06 20:24:40.883517089 +0000 UTC m=+44.599627732 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f2c17640-14e9-4b68-93a4-9183cbc2ccd0-metrics-tls") pod "dns-default-gqvcm" (UID: "f2c17640-14e9-4b68-93a4-9183cbc2ccd0") : secret "dns-default-metrics-tls" not found
Jul 06 20:24:37.121068 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:37.121037    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-fn7x6" event={"ID":"3cce0a2d-3fc6-4667-b38a-a0e800fbfe0b","Type":"ContainerStarted","Data":"dfa3a6de946b62733f0f86ffb57a42caf6eee625bdcfef9cb132b8468c062db7"}
Jul 06 20:24:37.123434 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:37.123410    2563 generic.go:356] "Generic (PLEG): container finished" podID="048dd56c-9989-44f2-a1d5-c3c50db6938c" containerID="a34d93e4b72cbaf6ce988d7558641d45a0eff2b9092464498c6c5967d8040c48" exitCode=0
Jul 06 20:24:37.123554 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:37.123454    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b7wcr" event={"ID":"048dd56c-9989-44f2-a1d5-c3c50db6938c","Type":"ContainerDied","Data":"a34d93e4b72cbaf6ce988d7558641d45a0eff2b9092464498c6c5967d8040c48"}
Jul 06 20:24:37.141716 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:37.141678    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-fn7x6" podStartSLOduration=0.968987722 podStartE2EDuration="4.141665786s" totalImagesPullingTime="3.172678064s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:24:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:24:37.140687625 +0000 UTC m=+40.856798286" watchObservedRunningTime="2026-07-06 20:24:37.141665786 +0000 UTC m=+40.857776455"
Jul 06 20:24:37.492591 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:37.492509    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:24:37.557032 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:37.556998    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:24:38.129053 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:38.129022    2563 generic.go:356] "Generic (PLEG): container finished" podID="048dd56c-9989-44f2-a1d5-c3c50db6938c" containerID="5cc5ce69b97da117d940c77a8dc7e9934dcb47c5ca4228c8c33a0e749c8c90de" exitCode=0
Jul 06 20:24:38.129195 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:38.129112    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b7wcr" event={"ID":"048dd56c-9989-44f2-a1d5-c3c50db6938c","Type":"ContainerDied","Data":"5cc5ce69b97da117d940c77a8dc7e9934dcb47c5ca4228c8c33a0e749c8c90de"}
Jul 06 20:24:39.133946 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:39.133912    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b7wcr" event={"ID":"048dd56c-9989-44f2-a1d5-c3c50db6938c","Type":"ContainerStarted","Data":"decc3e312e9d6de8bd1b9e8ccc44f569d9d1f581722e434743d54d9399ab7ac5"}
Jul 06 20:24:39.159077 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:39.159031    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-b7wcr" podStartSLOduration=8.20709205 podStartE2EDuration="42.159017131s" totalImagesPullingTime="33.951925081s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:23:57 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:24:39.157894601 +0000 UTC m=+42.874005260" watchObservedRunningTime="2026-07-06 20:24:39.159017131 +0000 UTC m=+42.875127790"
Jul 06 20:24:39.283130 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:39.283094    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:24:39.345098 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:39.345064    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:24:39.577465 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:39.577431    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:24:39.680992 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:39.680958    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:24:40.910216 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:40.910177    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f2c17640-14e9-4b68-93a4-9183cbc2ccd0-metrics-tls\") pod \"dns-default-gqvcm\" (UID: \"f2c17640-14e9-4b68-93a4-9183cbc2ccd0\") " pod="openshift-dns/dns-default-gqvcm"
Jul 06 20:24:40.910579 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:24:40.910299    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 06 20:24:40.910579 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:24:40.910352    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2c17640-14e9-4b68-93a4-9183cbc2ccd0-metrics-tls podName:f2c17640-14e9-4b68-93a4-9183cbc2ccd0 nodeName:}" failed. No retries permitted until 2026-07-06 20:24:48.910338749 +0000 UTC m=+52.626449387 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f2c17640-14e9-4b68-93a4-9183cbc2ccd0-metrics-tls") pod "dns-default-gqvcm" (UID: "f2c17640-14e9-4b68-93a4-9183cbc2ccd0") : secret "dns-default-metrics-tls" not found
Jul 06 20:24:48.968530 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:48.968486    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f2c17640-14e9-4b68-93a4-9183cbc2ccd0-metrics-tls\") pod \"dns-default-gqvcm\" (UID: \"f2c17640-14e9-4b68-93a4-9183cbc2ccd0\") " pod="openshift-dns/dns-default-gqvcm"
Jul 06 20:24:48.969040 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:24:48.968657    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 06 20:24:48.969040 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:24:48.968747    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2c17640-14e9-4b68-93a4-9183cbc2ccd0-metrics-tls podName:f2c17640-14e9-4b68-93a4-9183cbc2ccd0 nodeName:}" failed. No retries permitted until 2026-07-06 20:25:04.968712481 +0000 UTC m=+68.684823124 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f2c17640-14e9-4b68-93a4-9183cbc2ccd0-metrics-tls") pod "dns-default-gqvcm" (UID: "f2c17640-14e9-4b68-93a4-9183cbc2ccd0") : secret "dns-default-metrics-tls" not found
Jul 06 20:24:56.164193 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:56.164155    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-tcw8f"]
Jul 06 20:24:56.166664 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:56.166634    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-tcw8f"
Jul 06 20:24:56.168697 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:56.167869    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-2m7gh"]
Jul 06 20:24:56.169352 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:56.169330    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Jul 06 20:24:56.169721 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:56.169700    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Jul 06 20:24:56.169894 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:56.169879    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-rlkqb\""
Jul 06 20:24:56.170542 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:56.170519    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-68b757fb67-2m7gh"
Jul 06 20:24:56.172881 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:56.172865    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Jul 06 20:24:56.172960 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:56.172865    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Jul 06 20:24:56.175991 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:56.175975    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Jul 06 20:24:56.175991 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:56.175983    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-7jqzs\""
Jul 06 20:24:56.176327 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:56.176313    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Jul 06 20:24:56.181112 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:56.181094    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-tcw8f"]
Jul 06 20:24:56.182093 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:56.182077    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Jul 06 20:24:56.185439 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:56.185420    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-2m7gh"]
Jul 06 20:24:56.217245 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:56.217219    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/bb2a41aa-a9fc-4ec5-9b78-5e22c1fd301b-config\") pod \"console-operator-68b757fb67-2m7gh\" (UID: \"bb2a41aa-a9fc-4ec5-9b78-5e22c1fd301b\") " pod="openshift-console-operator/console-operator-68b757fb67-2m7gh"
Jul 06 20:24:56.217412 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:56.217261    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bb2a41aa-a9fc-4ec5-9b78-5e22c1fd301b-trusted-ca\") pod \"console-operator-68b757fb67-2m7gh\" (UID: \"bb2a41aa-a9fc-4ec5-9b78-5e22c1fd301b\") " pod="openshift-console-operator/console-operator-68b757fb67-2m7gh"
Jul 06 20:24:56.217412 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:56.217300    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzfdp\" (UniqueName: \"kubernetes.io/projected/bb2a41aa-a9fc-4ec5-9b78-5e22c1fd301b-kube-api-access-xzfdp\") pod \"console-operator-68b757fb67-2m7gh\" (UID: \"bb2a41aa-a9fc-4ec5-9b78-5e22c1fd301b\") " pod="openshift-console-operator/console-operator-68b757fb67-2m7gh"
Jul 06 20:24:56.217412 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:56.217367    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/bb2a41aa-a9fc-4ec5-9b78-5e22c1fd301b-serving-cert\") pod \"console-operator-68b757fb67-2m7gh\" (UID: \"bb2a41aa-a9fc-4ec5-9b78-5e22c1fd301b\") " pod="openshift-console-operator/console-operator-68b757fb67-2m7gh"
Jul 06 20:24:56.217586 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:56.217429    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvrp7\" (UniqueName: \"kubernetes.io/projected/c8b77a06-e378-4539-9e89-d0e6209b2da5-kube-api-access-xvrp7\") pod \"volume-data-source-validator-5d456fdbcd-tcw8f\" (UID: \"c8b77a06-e378-4539-9e89-d0e6209b2da5\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-tcw8f"
Jul 06 20:24:56.309260 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:56.309223    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:24:56.318223 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:56.318201    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/bb2a41aa-a9fc-4ec5-9b78-5e22c1fd301b-serving-cert\") pod \"console-operator-68b757fb67-2m7gh\" (UID: \"bb2a41aa-a9fc-4ec5-9b78-5e22c1fd301b\") " pod="openshift-console-operator/console-operator-68b757fb67-2m7gh"
Jul 06 20:24:56.318289 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:56.318239    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvrp7\" (UniqueName: \"kubernetes.io/projected/c8b77a06-e378-4539-9e89-d0e6209b2da5-kube-api-access-xvrp7\") pod \"volume-data-source-validator-5d456fdbcd-tcw8f\" (UID: \"c8b77a06-e378-4539-9e89-d0e6209b2da5\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-tcw8f"
Jul 06 20:24:56.318289 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:56.318283    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/bb2a41aa-a9fc-4ec5-9b78-5e22c1fd301b-config\") pod \"console-operator-68b757fb67-2m7gh\" (UID: \"bb2a41aa-a9fc-4ec5-9b78-5e22c1fd301b\") " pod="openshift-console-operator/console-operator-68b757fb67-2m7gh"
Jul 06 20:24:56.318363 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:56.318316    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bb2a41aa-a9fc-4ec5-9b78-5e22c1fd301b-trusted-ca\") pod \"console-operator-68b757fb67-2m7gh\" (UID: \"bb2a41aa-a9fc-4ec5-9b78-5e22c1fd301b\") " pod="openshift-console-operator/console-operator-68b757fb67-2m7gh"
Jul 06 20:24:56.318395 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:56.318369    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzfdp\" (UniqueName: \"kubernetes.io/projected/bb2a41aa-a9fc-4ec5-9b78-5e22c1fd301b-kube-api-access-xzfdp\") pod \"console-operator-68b757fb67-2m7gh\" (UID: \"bb2a41aa-a9fc-4ec5-9b78-5e22c1fd301b\") " pod="openshift-console-operator/console-operator-68b757fb67-2m7gh"
Jul 06 20:24:56.319187 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:56.319158    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/bb2a41aa-a9fc-4ec5-9b78-5e22c1fd301b-config\") pod \"console-operator-68b757fb67-2m7gh\" (UID: \"bb2a41aa-a9fc-4ec5-9b78-5e22c1fd301b\") " pod="openshift-console-operator/console-operator-68b757fb67-2m7gh"
Jul 06 20:24:56.319352 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:56.319333    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bb2a41aa-a9fc-4ec5-9b78-5e22c1fd301b-trusted-ca\") pod \"console-operator-68b757fb67-2m7gh\" (UID: \"bb2a41aa-a9fc-4ec5-9b78-5e22c1fd301b\") " pod="openshift-console-operator/console-operator-68b757fb67-2m7gh"
Jul 06 20:24:56.321456 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:56.321439    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/bb2a41aa-a9fc-4ec5-9b78-5e22c1fd301b-serving-cert\") pod \"console-operator-68b757fb67-2m7gh\" (UID: \"bb2a41aa-a9fc-4ec5-9b78-5e22c1fd301b\") " pod="openshift-console-operator/console-operator-68b757fb67-2m7gh"
Jul 06 20:24:56.332753 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:56.332208    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzfdp\" (UniqueName: \"kubernetes.io/projected/bb2a41aa-a9fc-4ec5-9b78-5e22c1fd301b-kube-api-access-xzfdp\") pod \"console-operator-68b757fb67-2m7gh\" (UID: \"bb2a41aa-a9fc-4ec5-9b78-5e22c1fd301b\") " pod="openshift-console-operator/console-operator-68b757fb67-2m7gh"
Jul 06 20:24:56.332863 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:56.332786    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvrp7\" (UniqueName: \"kubernetes.io/projected/c8b77a06-e378-4539-9e89-d0e6209b2da5-kube-api-access-xvrp7\") pod \"volume-data-source-validator-5d456fdbcd-tcw8f\" (UID: \"c8b77a06-e378-4539-9e89-d0e6209b2da5\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-tcw8f"
Jul 06 20:24:56.411538 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:56.411509    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:24:56.478619 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:56.478528    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-tcw8f"
Jul 06 20:24:56.483159 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:56.483140    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-68b757fb67-2m7gh"
Jul 06 20:24:56.604141 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:56.604109    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-tcw8f"]
Jul 06 20:24:56.607126 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:24:56.607100    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8b77a06_e378_4539_9e89_d0e6209b2da5.slice/crio-ce83d579034a519135ca0e72759ba6247e0a53bd86d35a22feab065b1698b253 WatchSource:0}: Error finding container ce83d579034a519135ca0e72759ba6247e0a53bd86d35a22feab065b1698b253: Status 404 returned error can't find the container with id ce83d579034a519135ca0e72759ba6247e0a53bd86d35a22feab065b1698b253
Jul 06 20:24:56.623267 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:56.623238    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-2m7gh"]
Jul 06 20:24:56.625912 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:24:56.625894    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb2a41aa_a9fc_4ec5_9b78_5e22c1fd301b.slice/crio-d0f251ac9db43c667d9da5d99b3ab91c1adceef116092b387a1ec734242c9efc WatchSource:0}: Error finding container d0f251ac9db43c667d9da5d99b3ab91c1adceef116092b387a1ec734242c9efc: Status 404 returned error can't find the container with id d0f251ac9db43c667d9da5d99b3ab91c1adceef116092b387a1ec734242c9efc
Jul 06 20:24:57.175980 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:57.175939    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-2m7gh" event={"ID":"bb2a41aa-a9fc-4ec5-9b78-5e22c1fd301b","Type":"ContainerStarted","Data":"d0f251ac9db43c667d9da5d99b3ab91c1adceef116092b387a1ec734242c9efc"}
Jul 06 20:24:57.177032 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:57.177004    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-tcw8f" event={"ID":"c8b77a06-e378-4539-9e89-d0e6209b2da5","Type":"ContainerStarted","Data":"ce83d579034a519135ca0e72759ba6247e0a53bd86d35a22feab065b1698b253"}
Jul 06 20:24:57.723892 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:57.723833    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7f79498b6c-bkcz7"]
Jul 06 20:24:57.737339 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:57.737313    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f79498b6c-bkcz7"
Jul 06 20:24:57.741254 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:57.741201    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 06 20:24:57.741565 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:57.741538    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 06 20:24:57.741800 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:57.741777    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 06 20:24:57.742943 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:57.741996    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-zmkh5\""
Jul 06 20:24:57.742943 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:57.742812    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7f79498b6c-bkcz7"]
Jul 06 20:24:57.748242 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:57.748219    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 06 20:24:57.830696 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:57.830656    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b27d84e3-9abb-4322-a9cb-c59f7d85ab10-registry-certificates\") pod \"image-registry-7f79498b6c-bkcz7\" (UID: \"b27d84e3-9abb-4322-a9cb-c59f7d85ab10\") " pod="openshift-image-registry/image-registry-7f79498b6c-bkcz7"
Jul 06 20:24:57.830904 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:57.830766    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b27d84e3-9abb-4322-a9cb-c59f7d85ab10-image-registry-private-configuration\") pod \"image-registry-7f79498b6c-bkcz7\" (UID: \"b27d84e3-9abb-4322-a9cb-c59f7d85ab10\") " pod="openshift-image-registry/image-registry-7f79498b6c-bkcz7"
Jul 06 20:24:57.830904 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:57.830795    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b27d84e3-9abb-4322-a9cb-c59f7d85ab10-ca-trust-extracted\") pod \"image-registry-7f79498b6c-bkcz7\" (UID: \"b27d84e3-9abb-4322-a9cb-c59f7d85ab10\") " pod="openshift-image-registry/image-registry-7f79498b6c-bkcz7"
Jul 06 20:24:57.830904 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:57.830818    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b27d84e3-9abb-4322-a9cb-c59f7d85ab10-bound-sa-token\") pod \"image-registry-7f79498b6c-bkcz7\" (UID: \"b27d84e3-9abb-4322-a9cb-c59f7d85ab10\") " pod="openshift-image-registry/image-registry-7f79498b6c-bkcz7"
Jul 06 20:24:57.830904 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:57.830851    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b27d84e3-9abb-4322-a9cb-c59f7d85ab10-registry-tls\") pod \"image-registry-7f79498b6c-bkcz7\" (UID: \"b27d84e3-9abb-4322-a9cb-c59f7d85ab10\") " pod="openshift-image-registry/image-registry-7f79498b6c-bkcz7"
Jul 06 20:24:57.831111 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:57.830935    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b27d84e3-9abb-4322-a9cb-c59f7d85ab10-installation-pull-secrets\") pod \"image-registry-7f79498b6c-bkcz7\" (UID: \"b27d84e3-9abb-4322-a9cb-c59f7d85ab10\") " pod="openshift-image-registry/image-registry-7f79498b6c-bkcz7"
Jul 06 20:24:57.831111 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:57.830958    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzwkh\" (UniqueName: \"kubernetes.io/projected/b27d84e3-9abb-4322-a9cb-c59f7d85ab10-kube-api-access-hzwkh\") pod \"image-registry-7f79498b6c-bkcz7\" (UID: \"b27d84e3-9abb-4322-a9cb-c59f7d85ab10\") " pod="openshift-image-registry/image-registry-7f79498b6c-bkcz7"
Jul 06 20:24:57.831111 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:57.830998    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b27d84e3-9abb-4322-a9cb-c59f7d85ab10-trusted-ca\") pod \"image-registry-7f79498b6c-bkcz7\" (UID: \"b27d84e3-9abb-4322-a9cb-c59f7d85ab10\") " pod="openshift-image-registry/image-registry-7f79498b6c-bkcz7"
Jul 06 20:24:57.932121 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:57.932085    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b27d84e3-9abb-4322-a9cb-c59f7d85ab10-trusted-ca\") pod \"image-registry-7f79498b6c-bkcz7\" (UID: \"b27d84e3-9abb-4322-a9cb-c59f7d85ab10\") " pod="openshift-image-registry/image-registry-7f79498b6c-bkcz7"
Jul 06 20:24:57.932310 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:57.932149    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b27d84e3-9abb-4322-a9cb-c59f7d85ab10-registry-certificates\") pod \"image-registry-7f79498b6c-bkcz7\" (UID: \"b27d84e3-9abb-4322-a9cb-c59f7d85ab10\") " pod="openshift-image-registry/image-registry-7f79498b6c-bkcz7"
Jul 06 20:24:57.932310 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:57.932199    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b27d84e3-9abb-4322-a9cb-c59f7d85ab10-image-registry-private-configuration\") pod \"image-registry-7f79498b6c-bkcz7\" (UID: \"b27d84e3-9abb-4322-a9cb-c59f7d85ab10\") " pod="openshift-image-registry/image-registry-7f79498b6c-bkcz7"
Jul 06 20:24:57.932310 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:57.932227    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b27d84e3-9abb-4322-a9cb-c59f7d85ab10-ca-trust-extracted\") pod \"image-registry-7f79498b6c-bkcz7\" (UID: \"b27d84e3-9abb-4322-a9cb-c59f7d85ab10\") " pod="openshift-image-registry/image-registry-7f79498b6c-bkcz7"
Jul 06 20:24:57.932310 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:57.932253    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b27d84e3-9abb-4322-a9cb-c59f7d85ab10-bound-sa-token\") pod \"image-registry-7f79498b6c-bkcz7\" (UID: \"b27d84e3-9abb-4322-a9cb-c59f7d85ab10\") " pod="openshift-image-registry/image-registry-7f79498b6c-bkcz7"
Jul 06 20:24:57.932310 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:57.932279    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b27d84e3-9abb-4322-a9cb-c59f7d85ab10-registry-tls\") pod \"image-registry-7f79498b6c-bkcz7\" (UID: \"b27d84e3-9abb-4322-a9cb-c59f7d85ab10\") " pod="openshift-image-registry/image-registry-7f79498b6c-bkcz7"
Jul 06 20:24:57.932556 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:57.932342    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b27d84e3-9abb-4322-a9cb-c59f7d85ab10-installation-pull-secrets\") pod \"image-registry-7f79498b6c-bkcz7\" (UID: \"b27d84e3-9abb-4322-a9cb-c59f7d85ab10\") " pod="openshift-image-registry/image-registry-7f79498b6c-bkcz7"
Jul 06 20:24:57.932556 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:57.932368    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzwkh\" (UniqueName: \"kubernetes.io/projected/b27d84e3-9abb-4322-a9cb-c59f7d85ab10-kube-api-access-hzwkh\") pod \"image-registry-7f79498b6c-bkcz7\" (UID: \"b27d84e3-9abb-4322-a9cb-c59f7d85ab10\") " pod="openshift-image-registry/image-registry-7f79498b6c-bkcz7"
Jul 06 20:24:57.932664 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:24:57.932561    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 06 20:24:57.932664 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:24:57.932593    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f79498b6c-bkcz7: secret "image-registry-tls" not found
Jul 06 20:24:57.932664 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:24:57.932654    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b27d84e3-9abb-4322-a9cb-c59f7d85ab10-registry-tls podName:b27d84e3-9abb-4322-a9cb-c59f7d85ab10 nodeName:}" failed. No retries permitted until 2026-07-06 20:24:58.432632455 +0000 UTC m=+62.148743111 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b27d84e3-9abb-4322-a9cb-c59f7d85ab10-registry-tls") pod "image-registry-7f79498b6c-bkcz7" (UID: "b27d84e3-9abb-4322-a9cb-c59f7d85ab10") : secret "image-registry-tls" not found
Jul 06 20:24:57.933122 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:57.933097    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b27d84e3-9abb-4322-a9cb-c59f7d85ab10-registry-certificates\") pod \"image-registry-7f79498b6c-bkcz7\" (UID: \"b27d84e3-9abb-4322-a9cb-c59f7d85ab10\") " pod="openshift-image-registry/image-registry-7f79498b6c-bkcz7"
Jul 06 20:24:57.933360 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:57.933337    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b27d84e3-9abb-4322-a9cb-c59f7d85ab10-trusted-ca\") pod \"image-registry-7f79498b6c-bkcz7\" (UID: \"b27d84e3-9abb-4322-a9cb-c59f7d85ab10\") " pod="openshift-image-registry/image-registry-7f79498b6c-bkcz7"
Jul 06 20:24:57.938044 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:57.938021    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b27d84e3-9abb-4322-a9cb-c59f7d85ab10-ca-trust-extracted\") pod \"image-registry-7f79498b6c-bkcz7\" (UID: \"b27d84e3-9abb-4322-a9cb-c59f7d85ab10\") " pod="openshift-image-registry/image-registry-7f79498b6c-bkcz7"
Jul 06 20:24:57.940347 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:57.940320    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b27d84e3-9abb-4322-a9cb-c59f7d85ab10-image-registry-private-configuration\") pod \"image-registry-7f79498b6c-bkcz7\" (UID: \"b27d84e3-9abb-4322-a9cb-c59f7d85ab10\") " pod="openshift-image-registry/image-registry-7f79498b6c-bkcz7"
Jul 06 20:24:57.940477 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:57.940448    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b27d84e3-9abb-4322-a9cb-c59f7d85ab10-installation-pull-secrets\") pod \"image-registry-7f79498b6c-bkcz7\" (UID: \"b27d84e3-9abb-4322-a9cb-c59f7d85ab10\") " pod="openshift-image-registry/image-registry-7f79498b6c-bkcz7"
Jul 06 20:24:57.941574 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:57.941549    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzwkh\" (UniqueName: \"kubernetes.io/projected/b27d84e3-9abb-4322-a9cb-c59f7d85ab10-kube-api-access-hzwkh\") pod \"image-registry-7f79498b6c-bkcz7\" (UID: \"b27d84e3-9abb-4322-a9cb-c59f7d85ab10\") " pod="openshift-image-registry/image-registry-7f79498b6c-bkcz7"
Jul 06 20:24:57.941797 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:57.941781    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b27d84e3-9abb-4322-a9cb-c59f7d85ab10-bound-sa-token\") pod \"image-registry-7f79498b6c-bkcz7\" (UID: \"b27d84e3-9abb-4322-a9cb-c59f7d85ab10\") " pod="openshift-image-registry/image-registry-7f79498b6c-bkcz7"
Jul 06 20:24:58.181272 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:58.181223    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-tcw8f" event={"ID":"c8b77a06-e378-4539-9e89-d0e6209b2da5","Type":"ContainerStarted","Data":"646d7b1c9b1bf2203150ddb7fd35c5db6914f9005efb91811708f8e034bd2399"}
Jul 06 20:24:58.205281 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:58.205233    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-tcw8f" podStartSLOduration=0.767928766 podStartE2EDuration="2.205219466s" totalImagesPullingTime="1.4372907s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:24:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:24:58.204962602 +0000 UTC m=+61.921073261" watchObservedRunningTime="2026-07-06 20:24:58.205219466 +0000 UTC m=+61.921330122"
Jul 06 20:24:58.308897 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:58.308828    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:24:58.380413 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:58.380379    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:24:58.437354 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:58.437312    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b27d84e3-9abb-4322-a9cb-c59f7d85ab10-registry-tls\") pod \"image-registry-7f79498b6c-bkcz7\" (UID: \"b27d84e3-9abb-4322-a9cb-c59f7d85ab10\") " pod="openshift-image-registry/image-registry-7f79498b6c-bkcz7"
Jul 06 20:24:58.437542 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:24:58.437480    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 06 20:24:58.437542 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:24:58.437503    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f79498b6c-bkcz7: secret "image-registry-tls" not found
Jul 06 20:24:58.437649 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:24:58.437583    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b27d84e3-9abb-4322-a9cb-c59f7d85ab10-registry-tls podName:b27d84e3-9abb-4322-a9cb-c59f7d85ab10 nodeName:}" failed. No retries permitted until 2026-07-06 20:24:59.437562268 +0000 UTC m=+63.153672920 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b27d84e3-9abb-4322-a9cb-c59f7d85ab10-registry-tls") pod "image-registry-7f79498b6c-bkcz7" (UID: "b27d84e3-9abb-4322-a9cb-c59f7d85ab10") : secret "image-registry-tls" not found
Jul 06 20:24:58.564642 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:58.564553    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:24:58.633107 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:58.633074    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:24:59.118200 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:59.118174    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5tk9w"
Jul 06 20:24:59.185562 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:59.185536    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-2m7gh_bb2a41aa-a9fc-4ec5-9b78-5e22c1fd301b/console-operator/0.log"
Jul 06 20:24:59.186052 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:59.185574    2563 generic.go:356] "Generic (PLEG): container finished" podID="bb2a41aa-a9fc-4ec5-9b78-5e22c1fd301b" containerID="2c445006176f451526c7dde73950f380340b537c03d73c7c731106ec16b88ccc" exitCode=255
Jul 06 20:24:59.186052 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:59.185653    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-2m7gh" event={"ID":"bb2a41aa-a9fc-4ec5-9b78-5e22c1fd301b","Type":"ContainerDied","Data":"2c445006176f451526c7dde73950f380340b537c03d73c7c731106ec16b88ccc"}
Jul 06 20:24:59.186052 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:59.185897    2563 scope.go:117] "RemoveContainer" containerID="2c445006176f451526c7dde73950f380340b537c03d73c7c731106ec16b88ccc"
Jul 06 20:24:59.285937 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:59.285893    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:24:59.360345 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:59.360311    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:24:59.445248 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:24:59.445213    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b27d84e3-9abb-4322-a9cb-c59f7d85ab10-registry-tls\") pod \"image-registry-7f79498b6c-bkcz7\" (UID: \"b27d84e3-9abb-4322-a9cb-c59f7d85ab10\") " pod="openshift-image-registry/image-registry-7f79498b6c-bkcz7"
Jul 06 20:24:59.445420 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:24:59.445339    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 06 20:24:59.445420 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:24:59.445353    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f79498b6c-bkcz7: secret "image-registry-tls" not found
Jul 06 20:24:59.445420 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:24:59.445400    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b27d84e3-9abb-4322-a9cb-c59f7d85ab10-registry-tls podName:b27d84e3-9abb-4322-a9cb-c59f7d85ab10 nodeName:}" failed. No retries permitted until 2026-07-06 20:25:01.445387939 +0000 UTC m=+65.161498578 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b27d84e3-9abb-4322-a9cb-c59f7d85ab10-registry-tls") pod "image-registry-7f79498b6c-bkcz7" (UID: "b27d84e3-9abb-4322-a9cb-c59f7d85ab10") : secret "image-registry-tls" not found
Jul 06 20:25:00.026773 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:00.026717    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-tl6cf"]
Jul 06 20:25:00.028489 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:00.028475    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-tl6cf"
Jul 06 20:25:00.030996 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:00.030965    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Jul 06 20:25:00.030996 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:00.030984    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Jul 06 20:25:00.031134 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:00.030986    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Jul 06 20:25:00.031189 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:00.031130    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-qssvn\""
Jul 06 20:25:00.031189 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:00.031135    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Jul 06 20:25:00.042161 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:00.042135    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-tl6cf"]
Jul 06 20:25:00.049233 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:00.049209    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mkr9\" (UniqueName: \"kubernetes.io/projected/209affe1-4e68-4730-afd0-975faf0bee4c-kube-api-access-4mkr9\") pod \"kube-storage-version-migrator-operator-846bf668f7-tl6cf\" (UID: \"209affe1-4e68-4730-afd0-975faf0bee4c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-tl6cf"
Jul 06 20:25:00.049363 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:00.049244    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/209affe1-4e68-4730-afd0-975faf0bee4c-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-tl6cf\" (UID: \"209affe1-4e68-4730-afd0-975faf0bee4c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-tl6cf"
Jul 06 20:25:00.049363 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:00.049331    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/209affe1-4e68-4730-afd0-975faf0bee4c-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-tl6cf\" (UID: \"209affe1-4e68-4730-afd0-975faf0bee4c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-tl6cf"
Jul 06 20:25:00.150207 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:00.150163    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/209affe1-4e68-4730-afd0-975faf0bee4c-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-tl6cf\" (UID: \"209affe1-4e68-4730-afd0-975faf0bee4c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-tl6cf"
Jul 06 20:25:00.150387 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:00.150286    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/209affe1-4e68-4730-afd0-975faf0bee4c-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-tl6cf\" (UID: \"209affe1-4e68-4730-afd0-975faf0bee4c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-tl6cf"
Jul 06 20:25:00.150387 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:00.150335    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mkr9\" (UniqueName: \"kubernetes.io/projected/209affe1-4e68-4730-afd0-975faf0bee4c-kube-api-access-4mkr9\") pod \"kube-storage-version-migrator-operator-846bf668f7-tl6cf\" (UID: \"209affe1-4e68-4730-afd0-975faf0bee4c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-tl6cf"
Jul 06 20:25:00.150839 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:00.150819    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/209affe1-4e68-4730-afd0-975faf0bee4c-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-tl6cf\" (UID: \"209affe1-4e68-4730-afd0-975faf0bee4c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-tl6cf"
Jul 06 20:25:00.152369 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:00.152349    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/209affe1-4e68-4730-afd0-975faf0bee4c-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-tl6cf\" (UID: \"209affe1-4e68-4730-afd0-975faf0bee4c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-tl6cf"
Jul 06 20:25:00.159353 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:00.159325    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mkr9\" (UniqueName: \"kubernetes.io/projected/209affe1-4e68-4730-afd0-975faf0bee4c-kube-api-access-4mkr9\") pod \"kube-storage-version-migrator-operator-846bf668f7-tl6cf\" (UID: \"209affe1-4e68-4730-afd0-975faf0bee4c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-tl6cf"
Jul 06 20:25:00.189088 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:00.189059    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-2m7gh_bb2a41aa-a9fc-4ec5-9b78-5e22c1fd301b/console-operator/1.log"
Jul 06 20:25:00.189473 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:00.189434    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-2m7gh_bb2a41aa-a9fc-4ec5-9b78-5e22c1fd301b/console-operator/0.log"
Jul 06 20:25:00.189473 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:00.189465    2563 generic.go:356] "Generic (PLEG): container finished" podID="bb2a41aa-a9fc-4ec5-9b78-5e22c1fd301b" containerID="fdb2317afffad2bb1d77ebb7539109a37fb34cbd1d02994eb4c149ca0b0e18b4" exitCode=255
Jul 06 20:25:00.189540 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:00.189493    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-2m7gh" event={"ID":"bb2a41aa-a9fc-4ec5-9b78-5e22c1fd301b","Type":"ContainerDied","Data":"fdb2317afffad2bb1d77ebb7539109a37fb34cbd1d02994eb4c149ca0b0e18b4"}
Jul 06 20:25:00.189540 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:00.189535    2563 scope.go:117] "RemoveContainer" containerID="2c445006176f451526c7dde73950f380340b537c03d73c7c731106ec16b88ccc"
Jul 06 20:25:00.189804 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:00.189786    2563 scope.go:117] "RemoveContainer" containerID="fdb2317afffad2bb1d77ebb7539109a37fb34cbd1d02994eb4c149ca0b0e18b4"
Jul 06 20:25:00.189987 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:25:00.189968    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-2m7gh_openshift-console-operator(bb2a41aa-a9fc-4ec5-9b78-5e22c1fd301b)\"" pod="openshift-console-operator/console-operator-68b757fb67-2m7gh" podUID="bb2a41aa-a9fc-4ec5-9b78-5e22c1fd301b"
Jul 06 20:25:00.339700 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:00.339657    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-tl6cf"
Jul 06 20:25:00.460035 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:00.460007    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-tl6cf"]
Jul 06 20:25:00.463148 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:25:00.463119    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod209affe1_4e68_4730_afd0_975faf0bee4c.slice/crio-194c2f24df29323baca5fded94c3b76dd1bf80e4556dc553aba904acfe190ac2 WatchSource:0}: Error finding container 194c2f24df29323baca5fded94c3b76dd1bf80e4556dc553aba904acfe190ac2: Status 404 returned error can't find the container with id 194c2f24df29323baca5fded94c3b76dd1bf80e4556dc553aba904acfe190ac2
Jul 06 20:25:00.783822 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:00.783715    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-hg45g"]
Jul 06 20:25:00.786632 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:00.786609    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hg45g"
Jul 06 20:25:00.789085 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:00.789063    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 06 20:25:00.789475 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:00.789459    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-k6n8v\""
Jul 06 20:25:00.789970 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:00.789954    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 06 20:25:00.790027 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:00.789969    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 06 20:25:00.790027 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:00.789989    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 06 20:25:00.799955 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:00.799929    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hg45g"]
Jul 06 20:25:00.813277 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:00.813245    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:25:00.856675 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:00.856639    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/abbed80c-bca8-49ce-91e5-692152f3d0e5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hg45g\" (UID: \"abbed80c-bca8-49ce-91e5-692152f3d0e5\") " pod="openshift-insights/insights-runtime-extractor-hg45g"
Jul 06 20:25:00.856675 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:00.856676    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/abbed80c-bca8-49ce-91e5-692152f3d0e5-crio-socket\") pod \"insights-runtime-extractor-hg45g\" (UID: \"abbed80c-bca8-49ce-91e5-692152f3d0e5\") " pod="openshift-insights/insights-runtime-extractor-hg45g"
Jul 06 20:25:00.856923 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:00.856701    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/abbed80c-bca8-49ce-91e5-692152f3d0e5-data-volume\") pod \"insights-runtime-extractor-hg45g\" (UID: \"abbed80c-bca8-49ce-91e5-692152f3d0e5\") " pod="openshift-insights/insights-runtime-extractor-hg45g"
Jul 06 20:25:00.856923 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:00.856817    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84t6d\" (UniqueName: \"kubernetes.io/projected/abbed80c-bca8-49ce-91e5-692152f3d0e5-kube-api-access-84t6d\") pod \"insights-runtime-extractor-hg45g\" (UID: \"abbed80c-bca8-49ce-91e5-692152f3d0e5\") " pod="openshift-insights/insights-runtime-extractor-hg45g"
Jul 06 20:25:00.856923 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:00.856847    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/abbed80c-bca8-49ce-91e5-692152f3d0e5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hg45g\" (UID: \"abbed80c-bca8-49ce-91e5-692152f3d0e5\") " pod="openshift-insights/insights-runtime-extractor-hg45g"
Jul 06 20:25:00.877287 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:00.877256    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:25:00.957490 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:00.957440    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/abbed80c-bca8-49ce-91e5-692152f3d0e5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hg45g\" (UID: \"abbed80c-bca8-49ce-91e5-692152f3d0e5\") " pod="openshift-insights/insights-runtime-extractor-hg45g"
Jul 06 20:25:00.957490 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:00.957490    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/abbed80c-bca8-49ce-91e5-692152f3d0e5-crio-socket\") pod \"insights-runtime-extractor-hg45g\" (UID: \"abbed80c-bca8-49ce-91e5-692152f3d0e5\") " pod="openshift-insights/insights-runtime-extractor-hg45g"
Jul 06 20:25:00.957670 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:00.957566    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/abbed80c-bca8-49ce-91e5-692152f3d0e5-crio-socket\") pod \"insights-runtime-extractor-hg45g\" (UID: \"abbed80c-bca8-49ce-91e5-692152f3d0e5\") " pod="openshift-insights/insights-runtime-extractor-hg45g"
Jul 06 20:25:00.957670 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:25:00.957598    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 06 20:25:00.957670 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:00.957614    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/abbed80c-bca8-49ce-91e5-692152f3d0e5-data-volume\") pod \"insights-runtime-extractor-hg45g\" (UID: \"abbed80c-bca8-49ce-91e5-692152f3d0e5\") " pod="openshift-insights/insights-runtime-extractor-hg45g"
Jul 06 20:25:00.957670 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:25:00.957669    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/abbed80c-bca8-49ce-91e5-692152f3d0e5-insights-runtime-extractor-tls podName:abbed80c-bca8-49ce-91e5-692152f3d0e5 nodeName:}" failed. No retries permitted until 2026-07-06 20:25:01.457650271 +0000 UTC m=+65.173760909 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/abbed80c-bca8-49ce-91e5-692152f3d0e5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hg45g" (UID: "abbed80c-bca8-49ce-91e5-692152f3d0e5") : secret "insights-runtime-extractor-tls" not found
Jul 06 20:25:00.957852 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:00.957697    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84t6d\" (UniqueName: \"kubernetes.io/projected/abbed80c-bca8-49ce-91e5-692152f3d0e5-kube-api-access-84t6d\") pod \"insights-runtime-extractor-hg45g\" (UID: \"abbed80c-bca8-49ce-91e5-692152f3d0e5\") " pod="openshift-insights/insights-runtime-extractor-hg45g"
Jul 06 20:25:00.957852 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:00.957715    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/abbed80c-bca8-49ce-91e5-692152f3d0e5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hg45g\" (UID: \"abbed80c-bca8-49ce-91e5-692152f3d0e5\") " pod="openshift-insights/insights-runtime-extractor-hg45g"
Jul 06 20:25:00.957852 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:25:00.957790    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 06 20:25:00.957852 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:25:00.957833    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/abbed80c-bca8-49ce-91e5-692152f3d0e5-kube-rbac-proxy-cm podName:abbed80c-bca8-49ce-91e5-692152f3d0e5 nodeName:}" failed. No retries permitted until 2026-07-06 20:25:01.457820498 +0000 UTC m=+65.173931142 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/abbed80c-bca8-49ce-91e5-692152f3d0e5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hg45g" (UID: "abbed80c-bca8-49ce-91e5-692152f3d0e5") : configmap "kube-rbac-proxy" not found
Jul 06 20:25:00.957977 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:00.957939    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/abbed80c-bca8-49ce-91e5-692152f3d0e5-data-volume\") pod \"insights-runtime-extractor-hg45g\" (UID: \"abbed80c-bca8-49ce-91e5-692152f3d0e5\") " pod="openshift-insights/insights-runtime-extractor-hg45g"
Jul 06 20:25:00.968271 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:00.968236    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84t6d\" (UniqueName: \"kubernetes.io/projected/abbed80c-bca8-49ce-91e5-692152f3d0e5-kube-api-access-84t6d\") pod \"insights-runtime-extractor-hg45g\" (UID: \"abbed80c-bca8-49ce-91e5-692152f3d0e5\") " pod="openshift-insights/insights-runtime-extractor-hg45g"
Jul 06 20:25:01.193458 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:01.193406    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-tl6cf" event={"ID":"209affe1-4e68-4730-afd0-975faf0bee4c","Type":"ContainerStarted","Data":"194c2f24df29323baca5fded94c3b76dd1bf80e4556dc553aba904acfe190ac2"}
Jul 06 20:25:01.195055 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:01.195032    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-2m7gh_bb2a41aa-a9fc-4ec5-9b78-5e22c1fd301b/console-operator/1.log"
Jul 06 20:25:01.195458 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:01.195437    2563 scope.go:117] "RemoveContainer" containerID="fdb2317afffad2bb1d77ebb7539109a37fb34cbd1d02994eb4c149ca0b0e18b4"
Jul 06 20:25:01.195654 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:25:01.195629    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-2m7gh_openshift-console-operator(bb2a41aa-a9fc-4ec5-9b78-5e22c1fd301b)\"" pod="openshift-console-operator/console-operator-68b757fb67-2m7gh" podUID="bb2a41aa-a9fc-4ec5-9b78-5e22c1fd301b"
Jul 06 20:25:01.462529 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:01.462433    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b27d84e3-9abb-4322-a9cb-c59f7d85ab10-registry-tls\") pod \"image-registry-7f79498b6c-bkcz7\" (UID: \"b27d84e3-9abb-4322-a9cb-c59f7d85ab10\") " pod="openshift-image-registry/image-registry-7f79498b6c-bkcz7"
Jul 06 20:25:01.462529 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:01.462506    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/abbed80c-bca8-49ce-91e5-692152f3d0e5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hg45g\" (UID: \"abbed80c-bca8-49ce-91e5-692152f3d0e5\") " pod="openshift-insights/insights-runtime-extractor-hg45g"
Jul 06 20:25:01.462796 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:01.462566    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/abbed80c-bca8-49ce-91e5-692152f3d0e5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hg45g\" (UID: \"abbed80c-bca8-49ce-91e5-692152f3d0e5\") " pod="openshift-insights/insights-runtime-extractor-hg45g"
Jul 06 20:25:01.462796 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:25:01.462598    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 06 20:25:01.462796 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:25:01.462621    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f79498b6c-bkcz7: secret "image-registry-tls" not found
Jul 06 20:25:01.462796 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:25:01.462671    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 06 20:25:01.462796 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:25:01.462683    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 06 20:25:01.462796 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:25:01.462686    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b27d84e3-9abb-4322-a9cb-c59f7d85ab10-registry-tls podName:b27d84e3-9abb-4322-a9cb-c59f7d85ab10 nodeName:}" failed. No retries permitted until 2026-07-06 20:25:05.462666577 +0000 UTC m=+69.178777239 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b27d84e3-9abb-4322-a9cb-c59f7d85ab10-registry-tls") pod "image-registry-7f79498b6c-bkcz7" (UID: "b27d84e3-9abb-4322-a9cb-c59f7d85ab10") : secret "image-registry-tls" not found
Jul 06 20:25:01.462796 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:25:01.462779    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/abbed80c-bca8-49ce-91e5-692152f3d0e5-kube-rbac-proxy-cm podName:abbed80c-bca8-49ce-91e5-692152f3d0e5 nodeName:}" failed. No retries permitted until 2026-07-06 20:25:02.462754834 +0000 UTC m=+66.178865488 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/abbed80c-bca8-49ce-91e5-692152f3d0e5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hg45g" (UID: "abbed80c-bca8-49ce-91e5-692152f3d0e5") : configmap "kube-rbac-proxy" not found
Jul 06 20:25:01.462796 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:25:01.462799    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/abbed80c-bca8-49ce-91e5-692152f3d0e5-insights-runtime-extractor-tls podName:abbed80c-bca8-49ce-91e5-692152f3d0e5 nodeName:}" failed. No retries permitted until 2026-07-06 20:25:02.462788089 +0000 UTC m=+66.178898741 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/abbed80c-bca8-49ce-91e5-692152f3d0e5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hg45g" (UID: "abbed80c-bca8-49ce-91e5-692152f3d0e5") : secret "insights-runtime-extractor-tls" not found
Jul 06 20:25:02.472810 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:02.472772    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/abbed80c-bca8-49ce-91e5-692152f3d0e5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hg45g\" (UID: \"abbed80c-bca8-49ce-91e5-692152f3d0e5\") " pod="openshift-insights/insights-runtime-extractor-hg45g"
Jul 06 20:25:02.473262 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:02.472891    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/abbed80c-bca8-49ce-91e5-692152f3d0e5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hg45g\" (UID: \"abbed80c-bca8-49ce-91e5-692152f3d0e5\") " pod="openshift-insights/insights-runtime-extractor-hg45g"
Jul 06 20:25:02.473262 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:25:02.472936    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 06 20:25:02.473262 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:25:02.473000    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 06 20:25:02.473262 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:25:02.473031    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/abbed80c-bca8-49ce-91e5-692152f3d0e5-kube-rbac-proxy-cm podName:abbed80c-bca8-49ce-91e5-692152f3d0e5 nodeName:}" failed. No retries permitted until 2026-07-06 20:25:04.473012256 +0000 UTC m=+68.189122895 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/abbed80c-bca8-49ce-91e5-692152f3d0e5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hg45g" (UID: "abbed80c-bca8-49ce-91e5-692152f3d0e5") : configmap "kube-rbac-proxy" not found
Jul 06 20:25:02.473262 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:25:02.473059    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/abbed80c-bca8-49ce-91e5-692152f3d0e5-insights-runtime-extractor-tls podName:abbed80c-bca8-49ce-91e5-692152f3d0e5 nodeName:}" failed. No retries permitted until 2026-07-06 20:25:04.473041796 +0000 UTC m=+68.189152452 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/abbed80c-bca8-49ce-91e5-692152f3d0e5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hg45g" (UID: "abbed80c-bca8-49ce-91e5-692152f3d0e5") : secret "insights-runtime-extractor-tls" not found
Jul 06 20:25:02.574285 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:02.574247    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e42d3c32-18ba-48dc-afb5-90ef7f0e2752-original-pull-secret\") pod \"global-pull-secret-syncer-nzf7w\" (UID: \"e42d3c32-18ba-48dc-afb5-90ef7f0e2752\") " pod="kube-system/global-pull-secret-syncer-nzf7w"
Jul 06 20:25:02.574473 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:02.574294    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b20534ab-b8c8-4c81-a0f4-0f98a738896e-metrics-certs\") pod \"network-metrics-daemon-gvd8v\" (UID: \"b20534ab-b8c8-4c81-a0f4-0f98a738896e\") " pod="openshift-multus/network-metrics-daemon-gvd8v"
Jul 06 20:25:02.576782 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:02.576761    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 06 20:25:02.576911 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:02.576836    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 06 20:25:02.584916 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:25:02.584881    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 06 20:25:02.585072 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:25:02.584991    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b20534ab-b8c8-4c81-a0f4-0f98a738896e-metrics-certs podName:b20534ab-b8c8-4c81-a0f4-0f98a738896e nodeName:}" failed. No retries permitted until 2026-07-06 20:26:06.584962999 +0000 UTC m=+130.301073687 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b20534ab-b8c8-4c81-a0f4-0f98a738896e-metrics-certs") pod "network-metrics-daemon-gvd8v" (UID: "b20534ab-b8c8-4c81-a0f4-0f98a738896e") : secret "metrics-daemon-secret" not found
Jul 06 20:25:02.587978 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:02.587953    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e42d3c32-18ba-48dc-afb5-90ef7f0e2752-original-pull-secret\") pod \"global-pull-secret-syncer-nzf7w\" (UID: \"e42d3c32-18ba-48dc-afb5-90ef7f0e2752\") " pod="kube-system/global-pull-secret-syncer-nzf7w"
Jul 06 20:25:02.641484 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:02.641445    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nzf7w"
Jul 06 20:25:02.675140 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:02.675097    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9wr4\" (UniqueName: \"kubernetes.io/projected/d2969391-caa9-4c3d-932c-3727d57b6ce8-kube-api-access-m9wr4\") pod \"network-check-target-tc5ls\" (UID: \"d2969391-caa9-4c3d-932c-3727d57b6ce8\") " pod="openshift-network-diagnostics/network-check-target-tc5ls"
Jul 06 20:25:02.677980 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:02.677959    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 06 20:25:02.688759 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:02.688715    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 06 20:25:02.699790 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:02.699758    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9wr4\" (UniqueName: \"kubernetes.io/projected/d2969391-caa9-4c3d-932c-3727d57b6ce8-kube-api-access-m9wr4\") pod \"network-check-target-tc5ls\" (UID: \"d2969391-caa9-4c3d-932c-3727d57b6ce8\") " pod="openshift-network-diagnostics/network-check-target-tc5ls"
Jul 06 20:25:02.748889 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:02.748814    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-n9568_1816531c-ac4a-4d49-96ac-234daa36ea91/dns-node-resolver/0.log"
Jul 06 20:25:02.901560 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:02.901534    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-nzf7w"]
Jul 06 20:25:02.904778 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:25:02.904752    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode42d3c32_18ba_48dc_afb5_90ef7f0e2752.slice/crio-a311b9285bcc930d248a9839eccb2e0dd6e566b7b51326e7a4a9b601fbc9c89c WatchSource:0}: Error finding container a311b9285bcc930d248a9839eccb2e0dd6e566b7b51326e7a4a9b601fbc9c89c: Status 404 returned error can't find the container with id a311b9285bcc930d248a9839eccb2e0dd6e566b7b51326e7a4a9b601fbc9c89c
Jul 06 20:25:02.951533 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:02.951502    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-hhd9j\""
Jul 06 20:25:02.960225 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:02.960202    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tc5ls"
Jul 06 20:25:03.089774 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:03.089721    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-tc5ls"]
Jul 06 20:25:03.092825 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:25:03.092802    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2969391_caa9_4c3d_932c_3727d57b6ce8.slice/crio-9aafe56584c0f04696fdc943b08ab182923a389171c0aa84bee82f72381f2849 WatchSource:0}: Error finding container 9aafe56584c0f04696fdc943b08ab182923a389171c0aa84bee82f72381f2849: Status 404 returned error can't find the container with id 9aafe56584c0f04696fdc943b08ab182923a389171c0aa84bee82f72381f2849
Jul 06 20:25:03.201804 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:03.201759    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-nzf7w" event={"ID":"e42d3c32-18ba-48dc-afb5-90ef7f0e2752","Type":"ContainerStarted","Data":"a311b9285bcc930d248a9839eccb2e0dd6e566b7b51326e7a4a9b601fbc9c89c"}
Jul 06 20:25:03.203215 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:03.203190    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-tl6cf" event={"ID":"209affe1-4e68-4730-afd0-975faf0bee4c","Type":"ContainerStarted","Data":"b35e6940b121a149690e9983ceddd51f72fbee4a49d7ab7514391201ea29584b"}
Jul 06 20:25:03.204194 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:03.204170    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-tc5ls" event={"ID":"d2969391-caa9-4c3d-932c-3727d57b6ce8","Type":"ContainerStarted","Data":"9aafe56584c0f04696fdc943b08ab182923a389171c0aa84bee82f72381f2849"}
Jul 06 20:25:03.224672 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:03.224560    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-tl6cf" podStartSLOduration=0.883541866 podStartE2EDuration="3.224538205s" totalImagesPullingTime="2.340996339s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:25:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:25:03.224104178 +0000 UTC m=+66.940214837" watchObservedRunningTime="2026-07-06 20:25:03.224538205 +0000 UTC m=+66.940648866"
Jul 06 20:25:03.547547 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:03.547515    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-2hwd9_ed41c82c-bbcc-4385-8f89-f6ff340c2e36/node-ca/0.log"
Jul 06 20:25:04.491805 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:04.491770    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/abbed80c-bca8-49ce-91e5-692152f3d0e5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hg45g\" (UID: \"abbed80c-bca8-49ce-91e5-692152f3d0e5\") " pod="openshift-insights/insights-runtime-extractor-hg45g"
Jul 06 20:25:04.492023 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:25:04.491891    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 06 20:25:04.492023 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:25:04.491953    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/abbed80c-bca8-49ce-91e5-692152f3d0e5-kube-rbac-proxy-cm podName:abbed80c-bca8-49ce-91e5-692152f3d0e5 nodeName:}" failed. No retries permitted until 2026-07-06 20:25:08.49193534 +0000 UTC m=+72.208045985 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/abbed80c-bca8-49ce-91e5-692152f3d0e5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hg45g" (UID: "abbed80c-bca8-49ce-91e5-692152f3d0e5") : configmap "kube-rbac-proxy" not found
Jul 06 20:25:04.492023 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:25:04.491968    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 06 20:25:04.492023 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:25:04.492003    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/abbed80c-bca8-49ce-91e5-692152f3d0e5-insights-runtime-extractor-tls podName:abbed80c-bca8-49ce-91e5-692152f3d0e5 nodeName:}" failed. No retries permitted until 2026-07-06 20:25:08.491993394 +0000 UTC m=+72.208104032 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/abbed80c-bca8-49ce-91e5-692152f3d0e5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hg45g" (UID: "abbed80c-bca8-49ce-91e5-692152f3d0e5") : secret "insights-runtime-extractor-tls" not found
Jul 06 20:25:04.492023 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:04.491888    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/abbed80c-bca8-49ce-91e5-692152f3d0e5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hg45g\" (UID: \"abbed80c-bca8-49ce-91e5-692152f3d0e5\") " pod="openshift-insights/insights-runtime-extractor-hg45g"
Jul 06 20:25:04.547202 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:04.547164    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-fn7x6_3cce0a2d-3fc6-4667-b38a-a0e800fbfe0b/serve-healthcheck-canary/0.log"
Jul 06 20:25:04.781227 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:04.781128    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:25:04.788964 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:04.788939    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-cf9kf"]
Jul 06 20:25:04.812482 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:04.812447    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-cf9kf"]
Jul 06 20:25:04.812658 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:04.812564    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-cf9kf"
Jul 06 20:25:04.815457 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:04.815400    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-czg6g\""
Jul 06 20:25:04.815580 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:04.815398    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Jul 06 20:25:04.816244 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:04.816223    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Jul 06 20:25:04.862305 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:04.862255    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:25:04.895040 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:04.895001    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dj57d\" (UniqueName: \"kubernetes.io/projected/775a6b7a-7141-4c1c-ad1f-1cb01de2aae9-kube-api-access-dj57d\") pod \"migrator-d57d58cc9-cf9kf\" (UID: \"775a6b7a-7141-4c1c-ad1f-1cb01de2aae9\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-cf9kf"
Jul 06 20:25:04.955305 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:04.955271    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:25:04.995394 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:04.995367    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f2c17640-14e9-4b68-93a4-9183cbc2ccd0-metrics-tls\") pod \"dns-default-gqvcm\" (UID: \"f2c17640-14e9-4b68-93a4-9183cbc2ccd0\") " pod="openshift-dns/dns-default-gqvcm"
Jul 06 20:25:04.995560 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:04.995417    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dj57d\" (UniqueName: \"kubernetes.io/projected/775a6b7a-7141-4c1c-ad1f-1cb01de2aae9-kube-api-access-dj57d\") pod \"migrator-d57d58cc9-cf9kf\" (UID: \"775a6b7a-7141-4c1c-ad1f-1cb01de2aae9\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-cf9kf"
Jul 06 20:25:04.995560 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:25:04.995512    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 06 20:25:04.995648 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:25:04.995569    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2c17640-14e9-4b68-93a4-9183cbc2ccd0-metrics-tls podName:f2c17640-14e9-4b68-93a4-9183cbc2ccd0 nodeName:}" failed. No retries permitted until 2026-07-06 20:25:36.995552896 +0000 UTC m=+100.711663534 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f2c17640-14e9-4b68-93a4-9183cbc2ccd0-metrics-tls") pod "dns-default-gqvcm" (UID: "f2c17640-14e9-4b68-93a4-9183cbc2ccd0") : secret "dns-default-metrics-tls" not found
Jul 06 20:25:05.004669 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:05.004639    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dj57d\" (UniqueName: \"kubernetes.io/projected/775a6b7a-7141-4c1c-ad1f-1cb01de2aae9-kube-api-access-dj57d\") pod \"migrator-d57d58cc9-cf9kf\" (UID: \"775a6b7a-7141-4c1c-ad1f-1cb01de2aae9\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-cf9kf"
Jul 06 20:25:05.124858 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:05.124824    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-cf9kf"
Jul 06 20:25:05.299786 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:05.299758    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-cf9kf"]
Jul 06 20:25:05.303191 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:25:05.303151    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod775a6b7a_7141_4c1c_ad1f_1cb01de2aae9.slice/crio-99240683ed205d83391acb9b50de3343e4959d0398f467621ce18e909a171f16 WatchSource:0}: Error finding container 99240683ed205d83391acb9b50de3343e4959d0398f467621ce18e909a171f16: Status 404 returned error can't find the container with id 99240683ed205d83391acb9b50de3343e4959d0398f467621ce18e909a171f16
Jul 06 20:25:05.500383 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:05.500297    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b27d84e3-9abb-4322-a9cb-c59f7d85ab10-registry-tls\") pod \"image-registry-7f79498b6c-bkcz7\" (UID: \"b27d84e3-9abb-4322-a9cb-c59f7d85ab10\") " pod="openshift-image-registry/image-registry-7f79498b6c-bkcz7"
Jul 06 20:25:05.500545 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:25:05.500468    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 06 20:25:05.500545 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:25:05.500491    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f79498b6c-bkcz7: secret "image-registry-tls" not found
Jul 06 20:25:05.500672 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:25:05.500559    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b27d84e3-9abb-4322-a9cb-c59f7d85ab10-registry-tls podName:b27d84e3-9abb-4322-a9cb-c59f7d85ab10 nodeName:}" failed. No retries permitted until 2026-07-06 20:25:13.500538349 +0000 UTC m=+77.216649002 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b27d84e3-9abb-4322-a9cb-c59f7d85ab10-registry-tls") pod "image-registry-7f79498b6c-bkcz7" (UID: "b27d84e3-9abb-4322-a9cb-c59f7d85ab10") : secret "image-registry-tls" not found
Jul 06 20:25:06.215607 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:06.215563    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-cf9kf" event={"ID":"775a6b7a-7141-4c1c-ad1f-1cb01de2aae9","Type":"ContainerStarted","Data":"99240683ed205d83391acb9b50de3343e4959d0398f467621ce18e909a171f16"}
Jul 06 20:25:06.484061 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:06.483977    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-68b757fb67-2m7gh"
Jul 06 20:25:06.484061 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:06.484042    2563 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-68b757fb67-2m7gh"
Jul 06 20:25:06.484453 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:06.484439    2563 scope.go:117] "RemoveContainer" containerID="fdb2317afffad2bb1d77ebb7539109a37fb34cbd1d02994eb4c149ca0b0e18b4"
Jul 06 20:25:06.484691 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:25:06.484664    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-2m7gh_openshift-console-operator(bb2a41aa-a9fc-4ec5-9b78-5e22c1fd301b)\"" pod="openshift-console-operator/console-operator-68b757fb67-2m7gh" podUID="bb2a41aa-a9fc-4ec5-9b78-5e22c1fd301b"
Jul 06 20:25:08.224508 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:08.224473    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-tc5ls" event={"ID":"d2969391-caa9-4c3d-932c-3727d57b6ce8","Type":"ContainerStarted","Data":"c91e2e453f8cb84ac997f840afadd67b0a6d304344ecebee7c581e343131acc3"}
Jul 06 20:25:08.225159 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:08.224676    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-tc5ls"
Jul 06 20:25:08.226036 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:08.226011    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-nzf7w" event={"ID":"e42d3c32-18ba-48dc-afb5-90ef7f0e2752","Type":"ContainerStarted","Data":"08e806539c62e24ed770c2a52a9ef6fda74963b99d35b9d11a4c6300616cb4d2"}
Jul 06 20:25:08.253626 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:08.253562    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-tc5ls" podStartSLOduration=66.338348371 podStartE2EDuration="1m11.253544032s" totalImagesPullingTime="4.915195661s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:23:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:25:08.249451265 +0000 UTC m=+71.965561925" watchObservedRunningTime="2026-07-06 20:25:08.253544032 +0000 UTC m=+71.969654692"
Jul 06 20:25:08.282999 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:08.282890    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-nzf7w" podStartSLOduration=65.176625889 podStartE2EDuration="1m10.282875133s" totalImagesPullingTime="5.106249244s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:23:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:25:08.282600821 +0000 UTC m=+71.998711482" watchObservedRunningTime="2026-07-06 20:25:08.282875133 +0000 UTC m=+71.998985799"
Jul 06 20:25:08.339972 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:08.339935    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:25:08.407778 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:08.407742    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:25:08.526680 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:08.526647    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/abbed80c-bca8-49ce-91e5-692152f3d0e5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hg45g\" (UID: \"abbed80c-bca8-49ce-91e5-692152f3d0e5\") " pod="openshift-insights/insights-runtime-extractor-hg45g"
Jul 06 20:25:08.526860 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:08.526703    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/abbed80c-bca8-49ce-91e5-692152f3d0e5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hg45g\" (UID: \"abbed80c-bca8-49ce-91e5-692152f3d0e5\") " pod="openshift-insights/insights-runtime-extractor-hg45g"
Jul 06 20:25:08.526860 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:25:08.526825    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 06 20:25:08.526860 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:25:08.526843    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 06 20:25:08.526961 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:25:08.526899    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/abbed80c-bca8-49ce-91e5-692152f3d0e5-insights-runtime-extractor-tls podName:abbed80c-bca8-49ce-91e5-692152f3d0e5 nodeName:}" failed. No retries permitted until 2026-07-06 20:25:16.526883281 +0000 UTC m=+80.242993932 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/abbed80c-bca8-49ce-91e5-692152f3d0e5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hg45g" (UID: "abbed80c-bca8-49ce-91e5-692152f3d0e5") : secret "insights-runtime-extractor-tls" not found
Jul 06 20:25:08.526961 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:25:08.526922    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/abbed80c-bca8-49ce-91e5-692152f3d0e5-kube-rbac-proxy-cm podName:abbed80c-bca8-49ce-91e5-692152f3d0e5 nodeName:}" failed. No retries permitted until 2026-07-06 20:25:16.526908416 +0000 UTC m=+80.243019058 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/abbed80c-bca8-49ce-91e5-692152f3d0e5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hg45g" (UID: "abbed80c-bca8-49ce-91e5-692152f3d0e5") : configmap "kube-rbac-proxy" not found
Jul 06 20:25:08.784421 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:08.784381    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:25:08.889665 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:08.889628    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:25:09.061958 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:09.061931    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:25:09.176189 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:09.176160    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:25:09.230885 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:09.230851    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-cf9kf" event={"ID":"775a6b7a-7141-4c1c-ad1f-1cb01de2aae9","Type":"ContainerStarted","Data":"65cac4462a3e5c9a2a75f44421d3e6a650ff2e9601c0bfb8a002ad1362136e66"}
Jul 06 20:25:09.231333 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:09.230915    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-cf9kf" event={"ID":"775a6b7a-7141-4c1c-ad1f-1cb01de2aae9","Type":"ContainerStarted","Data":"b92169d8b5107e2d08f7b8cc76cfe967e613589f41bf11cc3bc6f5b1c586b8fc"}
Jul 06 20:25:09.255208 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:09.255163    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-cf9kf" podStartSLOduration=1.512436135 podStartE2EDuration="5.255148158s" totalImagesPullingTime="3.742712023s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:25:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:25:09.254166898 +0000 UTC m=+72.970277558" watchObservedRunningTime="2026-07-06 20:25:09.255148158 +0000 UTC m=+72.971258818"
Jul 06 20:25:09.400505 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:09.400468    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:25:09.515694 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:09.515666    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:25:10.636650 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:10.636606    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-758f599c5-mw52s"]
Jul 06 20:25:10.638797 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:10.638782    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-758f599c5-mw52s"
Jul 06 20:25:10.643019 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:10.642994    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Jul 06 20:25:10.643127 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:10.642994    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Jul 06 20:25:10.643127 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:10.643034    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Jul 06 20:25:10.643127 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:10.643049    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-g68p2\""
Jul 06 20:25:10.643127 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:10.643055    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Jul 06 20:25:10.650550 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:10.650530    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-758f599c5-mw52s"]
Jul 06 20:25:10.683813 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:10.683788    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:25:10.744209 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:10.744182    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tl2w\" (UniqueName: \"kubernetes.io/projected/b065d6e1-1c81-4250-b367-ad225d6a71b4-kube-api-access-5tl2w\") pod \"service-ca-758f599c5-mw52s\" (UID: \"b065d6e1-1c81-4250-b367-ad225d6a71b4\") " pod="openshift-service-ca/service-ca-758f599c5-mw52s"
Jul 06 20:25:10.744334 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:10.744218    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/b065d6e1-1c81-4250-b367-ad225d6a71b4-signing-key\") pod \"service-ca-758f599c5-mw52s\" (UID: \"b065d6e1-1c81-4250-b367-ad225d6a71b4\") " pod="openshift-service-ca/service-ca-758f599c5-mw52s"
Jul 06 20:25:10.744334 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:10.744279    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/b065d6e1-1c81-4250-b367-ad225d6a71b4-signing-cabundle\") pod \"service-ca-758f599c5-mw52s\" (UID: \"b065d6e1-1c81-4250-b367-ad225d6a71b4\") " pod="openshift-service-ca/service-ca-758f599c5-mw52s"
Jul 06 20:25:10.760085 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:10.760056    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:25:10.844981 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:10.844951    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tl2w\" (UniqueName: \"kubernetes.io/projected/b065d6e1-1c81-4250-b367-ad225d6a71b4-kube-api-access-5tl2w\") pod \"service-ca-758f599c5-mw52s\" (UID: \"b065d6e1-1c81-4250-b367-ad225d6a71b4\") " pod="openshift-service-ca/service-ca-758f599c5-mw52s"
Jul 06 20:25:10.844981 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:10.844982    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/b065d6e1-1c81-4250-b367-ad225d6a71b4-signing-key\") pod \"service-ca-758f599c5-mw52s\" (UID: \"b065d6e1-1c81-4250-b367-ad225d6a71b4\") " pod="openshift-service-ca/service-ca-758f599c5-mw52s"
Jul 06 20:25:10.845177 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:10.845018    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/b065d6e1-1c81-4250-b367-ad225d6a71b4-signing-cabundle\") pod \"service-ca-758f599c5-mw52s\" (UID: \"b065d6e1-1c81-4250-b367-ad225d6a71b4\") " pod="openshift-service-ca/service-ca-758f599c5-mw52s"
Jul 06 20:25:10.845623 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:10.845592    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/b065d6e1-1c81-4250-b367-ad225d6a71b4-signing-cabundle\") pod \"service-ca-758f599c5-mw52s\" (UID: \"b065d6e1-1c81-4250-b367-ad225d6a71b4\") " pod="openshift-service-ca/service-ca-758f599c5-mw52s"
Jul 06 20:25:10.847265 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:10.847239    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/b065d6e1-1c81-4250-b367-ad225d6a71b4-signing-key\") pod \"service-ca-758f599c5-mw52s\" (UID: \"b065d6e1-1c81-4250-b367-ad225d6a71b4\") " pod="openshift-service-ca/service-ca-758f599c5-mw52s"
Jul 06 20:25:10.853593 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:10.853573    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tl2w\" (UniqueName: \"kubernetes.io/projected/b065d6e1-1c81-4250-b367-ad225d6a71b4-kube-api-access-5tl2w\") pod \"service-ca-758f599c5-mw52s\" (UID: \"b065d6e1-1c81-4250-b367-ad225d6a71b4\") " pod="openshift-service-ca/service-ca-758f599c5-mw52s"
Jul 06 20:25:10.946746 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:10.946628    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-758f599c5-mw52s"
Jul 06 20:25:11.078927 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:11.078901    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-758f599c5-mw52s"]
Jul 06 20:25:11.082148 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:25:11.082119    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb065d6e1_1c81_4250_b367_ad225d6a71b4.slice/crio-11455d427a098a1aebfef4cf91658174b886233fe159efb27a937940d8ea83fc WatchSource:0}: Error finding container 11455d427a098a1aebfef4cf91658174b886233fe159efb27a937940d8ea83fc: Status 404 returned error can't find the container with id 11455d427a098a1aebfef4cf91658174b886233fe159efb27a937940d8ea83fc
Jul 06 20:25:11.240121 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:11.240036    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-758f599c5-mw52s" event={"ID":"b065d6e1-1c81-4250-b367-ad225d6a71b4","Type":"ContainerStarted","Data":"11455d427a098a1aebfef4cf91658174b886233fe159efb27a937940d8ea83fc"}
Jul 06 20:25:11.718561 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:11.718521    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:25:13.247561 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:13.247467    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-758f599c5-mw52s" event={"ID":"b065d6e1-1c81-4250-b367-ad225d6a71b4","Type":"ContainerStarted","Data":"666a123d762867bc71939800ddd101ed0c98b6c304e253d5f035e46a68845d04"}
Jul 06 20:25:13.271009 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:13.270956    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-758f599c5-mw52s" podStartSLOduration=1.459455334 podStartE2EDuration="3.27094043s" totalImagesPullingTime="1.811485096s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:25:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:25:13.269813324 +0000 UTC m=+76.985923984" watchObservedRunningTime="2026-07-06 20:25:13.27094043 +0000 UTC m=+76.987051100"
Jul 06 20:25:13.568970 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:13.568942    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b27d84e3-9abb-4322-a9cb-c59f7d85ab10-registry-tls\") pod \"image-registry-7f79498b6c-bkcz7\" (UID: \"b27d84e3-9abb-4322-a9cb-c59f7d85ab10\") " pod="openshift-image-registry/image-registry-7f79498b6c-bkcz7"
Jul 06 20:25:13.569120 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:25:13.569048    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 06 20:25:13.569120 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:25:13.569062    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f79498b6c-bkcz7: secret "image-registry-tls" not found
Jul 06 20:25:13.569120 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:25:13.569107    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b27d84e3-9abb-4322-a9cb-c59f7d85ab10-registry-tls podName:b27d84e3-9abb-4322-a9cb-c59f7d85ab10 nodeName:}" failed. No retries permitted until 2026-07-06 20:25:29.569094649 +0000 UTC m=+93.285205288 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b27d84e3-9abb-4322-a9cb-c59f7d85ab10-registry-tls") pod "image-registry-7f79498b6c-bkcz7" (UID: "b27d84e3-9abb-4322-a9cb-c59f7d85ab10") : secret "image-registry-tls" not found
Jul 06 20:25:13.994720 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:13.994618    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:25:14.332948 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:14.332896    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:25:14.639093 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:14.639001    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:25:14.709922 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:14.709891    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:25:14.780840 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:14.780811    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:25:14.880699 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:14.880646    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:25:14.982832 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:14.982737    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:25:15.083017 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:15.082983    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:25:15.857553 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:15.857515    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:25:15.940955 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:15.940919    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:25:16.430075 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:16.430048    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:25:16.594807 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:16.594770    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/abbed80c-bca8-49ce-91e5-692152f3d0e5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hg45g\" (UID: \"abbed80c-bca8-49ce-91e5-692152f3d0e5\") " pod="openshift-insights/insights-runtime-extractor-hg45g"
Jul 06 20:25:16.594987 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:16.594886    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/abbed80c-bca8-49ce-91e5-692152f3d0e5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hg45g\" (UID: \"abbed80c-bca8-49ce-91e5-692152f3d0e5\") " pod="openshift-insights/insights-runtime-extractor-hg45g"
Jul 06 20:25:16.595059 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:25:16.595040    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 06 20:25:16.595144 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:25:16.595131    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/abbed80c-bca8-49ce-91e5-692152f3d0e5-insights-runtime-extractor-tls podName:abbed80c-bca8-49ce-91e5-692152f3d0e5 nodeName:}" failed. No retries permitted until 2026-07-06 20:25:32.595109587 +0000 UTC m=+96.311220245 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/abbed80c-bca8-49ce-91e5-692152f3d0e5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hg45g" (UID: "abbed80c-bca8-49ce-91e5-692152f3d0e5") : secret "insights-runtime-extractor-tls" not found
Jul 06 20:25:16.595389 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:16.595371    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/abbed80c-bca8-49ce-91e5-692152f3d0e5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hg45g\" (UID: \"abbed80c-bca8-49ce-91e5-692152f3d0e5\") " pod="openshift-insights/insights-runtime-extractor-hg45g"
Jul 06 20:25:16.634831 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:16.634788    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:25:16.844132 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:16.844085    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:25:17.250273 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:17.250184    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:25:17.834343 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:17.834307    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:25:18.027972 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:18.027921    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:25:18.196903 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:18.196739    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:25:18.441575 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:18.441545    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:25:18.540315 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:18.540235    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:25:18.828751 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:18.828705    2563 scope.go:117] "RemoveContainer" containerID="fdb2317afffad2bb1d77ebb7539109a37fb34cbd1d02994eb4c149ca0b0e18b4"
Jul 06 20:25:19.032286 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:19.032258    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:25:19.247467 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:19.247368    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:25:19.271227 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:19.271200    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-2m7gh_bb2a41aa-a9fc-4ec5-9b78-5e22c1fd301b/console-operator/1.log"
Jul 06 20:25:19.271371 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:19.271279    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-2m7gh" event={"ID":"bb2a41aa-a9fc-4ec5-9b78-5e22c1fd301b","Type":"ContainerStarted","Data":"c01ec9433953c4cdf042e2dc38ec6bca2290e29188c5cdf9795b934d20c18581"}
Jul 06 20:25:19.271690 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:19.271665    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-68b757fb67-2m7gh"
Jul 06 20:25:19.278073 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:19.278051    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-68b757fb67-2m7gh"
Jul 06 20:25:19.292429 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:19.292369    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-68b757fb67-2m7gh" podStartSLOduration=20.94889705 podStartE2EDuration="23.292352363s" totalImagesPullingTime="2.343455313s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:24:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:25:19.291306726 +0000 UTC m=+83.007417386" watchObservedRunningTime="2026-07-06 20:25:19.292352363 +0000 UTC m=+83.008463081"
Jul 06 20:25:19.447847 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:19.447807    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:25:19.553068 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:19.552980    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:25:19.651164 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:19.651125    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:25:19.753620 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:19.753591    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:25:19.854457 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:19.854428    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:25:19.954095 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:19.954059    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:25:20.115428 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:20.115339    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:25:20.236379 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:20.236349    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:25:20.338718 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:20.338687    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:25:20.438132 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:20.438037    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:25:20.539889 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:20.539853    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:25:20.639913 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:20.639879    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:25:20.740140 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:20.740054    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:25:20.840319 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:20.840279    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:25:21.029925 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:21.029826    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:25:21.132656 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:21.132616    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:25:21.328666 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:21.328617    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:25:21.431623 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:21.431589    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:25:21.641324 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:21.641239    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:25:21.949196 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:21.949113    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:25:22.050546 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:22.050513    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:25:22.114803 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:22.114772    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:25:22.839622 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:22.839587    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:25:23.031330 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:23.031293    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:25:23.448681 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:23.448609    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:25:23.634357 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:23.634326    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:25:24.229248 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:24.229212    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:25:24.832950 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:24.832918    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:25:25.024160 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:25.024131    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:25:25.227874 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:25.227791    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:25:25.433333 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:25.433300    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:25:26.202770 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:26.202737    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:25:26.556267 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:26.556235    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:25:29.599773 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:29.599741    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b27d84e3-9abb-4322-a9cb-c59f7d85ab10-registry-tls\") pod \"image-registry-7f79498b6c-bkcz7\" (UID: \"b27d84e3-9abb-4322-a9cb-c59f7d85ab10\") " pod="openshift-image-registry/image-registry-7f79498b6c-bkcz7"
Jul 06 20:25:29.602133 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:29.602111    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b27d84e3-9abb-4322-a9cb-c59f7d85ab10-registry-tls\") pod \"image-registry-7f79498b6c-bkcz7\" (UID: \"b27d84e3-9abb-4322-a9cb-c59f7d85ab10\") " pod="openshift-image-registry/image-registry-7f79498b6c-bkcz7"
Jul 06 20:25:29.850196 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:29.850110    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f79498b6c-bkcz7"
Jul 06 20:25:29.981531 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:29.981499    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7f79498b6c-bkcz7"]
Jul 06 20:25:29.984590 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:25:29.984566    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb27d84e3_9abb_4322_a9cb_c59f7d85ab10.slice/crio-d192c9b7ad7d2572c1cb907526628589c6015d95c224e22947e0e6ee2bc1a069 WatchSource:0}: Error finding container d192c9b7ad7d2572c1cb907526628589c6015d95c224e22947e0e6ee2bc1a069: Status 404 returned error can't find the container with id d192c9b7ad7d2572c1cb907526628589c6015d95c224e22947e0e6ee2bc1a069
Jul 06 20:25:30.306002 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:30.305968    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f79498b6c-bkcz7" event={"ID":"b27d84e3-9abb-4322-a9cb-c59f7d85ab10","Type":"ContainerStarted","Data":"0307c428fe6aab1962abd7880ad58ebec0734b8c8b7120c443b02613bb54e54d"}
Jul 06 20:25:30.306167 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:30.306026    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f79498b6c-bkcz7" event={"ID":"b27d84e3-9abb-4322-a9cb-c59f7d85ab10","Type":"ContainerStarted","Data":"d192c9b7ad7d2572c1cb907526628589c6015d95c224e22947e0e6ee2bc1a069"}
Jul 06 20:25:30.306167 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:30.306114    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7f79498b6c-bkcz7"
Jul 06 20:25:30.325976 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:30.325927    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7f79498b6c-bkcz7" podStartSLOduration=33.325914049 podStartE2EDuration="33.325914049s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:24:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:25:30.325175462 +0000 UTC m=+94.041286123" watchObservedRunningTime="2026-07-06 20:25:30.325914049 +0000 UTC m=+94.042024709"
Jul 06 20:25:31.479617 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:31.479586    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:25:31.578123 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:31.578092    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:25:31.656111 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:31.656078    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:25:32.030229 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:32.030192    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:25:32.625127 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:32.625098    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/abbed80c-bca8-49ce-91e5-692152f3d0e5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hg45g\" (UID: \"abbed80c-bca8-49ce-91e5-692152f3d0e5\") " pod="openshift-insights/insights-runtime-extractor-hg45g"
Jul 06 20:25:32.627240 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:32.627218    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/abbed80c-bca8-49ce-91e5-692152f3d0e5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hg45g\" (UID: \"abbed80c-bca8-49ce-91e5-692152f3d0e5\") " pod="openshift-insights/insights-runtime-extractor-hg45g"
Jul 06 20:25:32.895404 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:32.895321    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hg45g"
Jul 06 20:25:33.022410 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:33.022381    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hg45g"]
Jul 06 20:25:33.028177 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:25:33.028146    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabbed80c_bca8_49ce_91e5_692152f3d0e5.slice/crio-916c3100cb214e90a2ebffd80fffd3289ab84e336e0f7d6fe95f0b478aa6c44f WatchSource:0}: Error finding container 916c3100cb214e90a2ebffd80fffd3289ab84e336e0f7d6fe95f0b478aa6c44f: Status 404 returned error can't find the container with id 916c3100cb214e90a2ebffd80fffd3289ab84e336e0f7d6fe95f0b478aa6c44f
Jul 06 20:25:33.316439 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:33.316410    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hg45g" event={"ID":"abbed80c-bca8-49ce-91e5-692152f3d0e5","Type":"ContainerStarted","Data":"916c3100cb214e90a2ebffd80fffd3289ab84e336e0f7d6fe95f0b478aa6c44f"}
Jul 06 20:25:33.765325 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:33.765151    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7f79498b6c-bkcz7"]
Jul 06 20:25:33.798243 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:33.798179    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:25:33.990483 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:33.990363    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:25:34.130080 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:34.130025    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:25:34.238997 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:34.238954    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:25:35.560213 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:35.560095    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:25:35.661318 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:35.661246    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:25:35.761024 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:35.760996    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:25:35.826164 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:35.826094    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:25:37.061399 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:37.061370    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f2c17640-14e9-4b68-93a4-9183cbc2ccd0-metrics-tls\") pod \"dns-default-gqvcm\" (UID: \"f2c17640-14e9-4b68-93a4-9183cbc2ccd0\") " pod="openshift-dns/dns-default-gqvcm"
Jul 06 20:25:37.063501 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:37.063482    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f2c17640-14e9-4b68-93a4-9183cbc2ccd0-metrics-tls\") pod \"dns-default-gqvcm\" (UID: \"f2c17640-14e9-4b68-93a4-9183cbc2ccd0\") " pod="openshift-dns/dns-default-gqvcm"
Jul 06 20:25:37.330413 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:37.330386    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hg45g" event={"ID":"abbed80c-bca8-49ce-91e5-692152f3d0e5","Type":"ContainerStarted","Data":"792a361408347de0b9cdba35d3f8e6d7cfd7b3ba5acbd3ba38ae3a42c6f818b9"}
Jul 06 20:25:37.349996 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:37.349971    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-mqhkz\""
Jul 06 20:25:37.358808 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:37.358788    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-gqvcm"
Jul 06 20:25:37.510125 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:37.510097    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-gqvcm"]
Jul 06 20:25:37.512606 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:25:37.512583    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2c17640_14e9_4b68_93a4_9183cbc2ccd0.slice/crio-9b036771cd04b7744c8e20669685a682cd38e7aeda11478da7a77cd8cf90a409 WatchSource:0}: Error finding container 9b036771cd04b7744c8e20669685a682cd38e7aeda11478da7a77cd8cf90a409: Status 404 returned error can't find the container with id 9b036771cd04b7744c8e20669685a682cd38e7aeda11478da7a77cd8cf90a409
Jul 06 20:25:37.557448 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:37.557414    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:25:37.644139 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:37.644054    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:25:38.335941 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:38.335907    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hg45g" event={"ID":"abbed80c-bca8-49ce-91e5-692152f3d0e5","Type":"ContainerStarted","Data":"7d2cf1f644ee5a3d19932038126d92c33762a18f5633569e982bcc93e0753d7a"}
Jul 06 20:25:38.337126 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:38.337067    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gqvcm" event={"ID":"f2c17640-14e9-4b68-93a4-9183cbc2ccd0","Type":"ContainerStarted","Data":"9b036771cd04b7744c8e20669685a682cd38e7aeda11478da7a77cd8cf90a409"}
Jul 06 20:25:38.677739 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:38.677651    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:25:38.730292 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:38.730258    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-qphwl"]
Jul 06 20:25:38.733751 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:38.733673    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-qphwl"
Jul 06 20:25:38.736771 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:38.736749    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 06 20:25:38.737881 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:38.737721    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 06 20:25:38.738036 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:38.738013    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 06 20:25:38.738451 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:38.738421    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 06 20:25:38.738942 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:38.738921    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 06 20:25:38.739054 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:38.738959    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-h26dt\""
Jul 06 20:25:38.748415 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:38.748394    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-qphwl"]
Jul 06 20:25:38.770949 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:38.770903    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:25:38.775903 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:38.775882    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zm48k\" (UniqueName: \"kubernetes.io/projected/b33528c3-08cc-4595-aa46-8a72187a353d-kube-api-access-zm48k\") pod \"prometheus-operator-567f7b8b7c-qphwl\" (UID: \"b33528c3-08cc-4595-aa46-8a72187a353d\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-qphwl"
Jul 06 20:25:38.776062 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:38.775919    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b33528c3-08cc-4595-aa46-8a72187a353d-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-qphwl\" (UID: \"b33528c3-08cc-4595-aa46-8a72187a353d\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-qphwl"
Jul 06 20:25:38.776062 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:38.775954    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b33528c3-08cc-4595-aa46-8a72187a353d-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-qphwl\" (UID: \"b33528c3-08cc-4595-aa46-8a72187a353d\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-qphwl"
Jul 06 20:25:38.776062 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:38.776019    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/b33528c3-08cc-4595-aa46-8a72187a353d-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-qphwl\" (UID: \"b33528c3-08cc-4595-aa46-8a72187a353d\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-qphwl"
Jul 06 20:25:38.876700 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:38.876652    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b33528c3-08cc-4595-aa46-8a72187a353d-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-qphwl\" (UID: \"b33528c3-08cc-4595-aa46-8a72187a353d\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-qphwl"
Jul 06 20:25:38.876909 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:38.876749    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/b33528c3-08cc-4595-aa46-8a72187a353d-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-qphwl\" (UID: \"b33528c3-08cc-4595-aa46-8a72187a353d\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-qphwl"
Jul 06 20:25:38.876909 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:38.876851    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zm48k\" (UniqueName: \"kubernetes.io/projected/b33528c3-08cc-4595-aa46-8a72187a353d-kube-api-access-zm48k\") pod \"prometheus-operator-567f7b8b7c-qphwl\" (UID: \"b33528c3-08cc-4595-aa46-8a72187a353d\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-qphwl"
Jul 06 20:25:38.876909 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:38.876876    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b33528c3-08cc-4595-aa46-8a72187a353d-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-qphwl\" (UID: \"b33528c3-08cc-4595-aa46-8a72187a353d\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-qphwl"
Jul 06 20:25:38.876909 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:25:38.876881    2563 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Jul 06 20:25:38.877118 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:25:38.876954    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b33528c3-08cc-4595-aa46-8a72187a353d-prometheus-operator-tls podName:b33528c3-08cc-4595-aa46-8a72187a353d nodeName:}" failed. No retries permitted until 2026-07-06 20:25:39.376934171 +0000 UTC m=+103.093044822 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/b33528c3-08cc-4595-aa46-8a72187a353d-prometheus-operator-tls") pod "prometheus-operator-567f7b8b7c-qphwl" (UID: "b33528c3-08cc-4595-aa46-8a72187a353d") : secret "prometheus-operator-tls" not found
Jul 06 20:25:38.877433 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:38.877407    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b33528c3-08cc-4595-aa46-8a72187a353d-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-qphwl\" (UID: \"b33528c3-08cc-4595-aa46-8a72187a353d\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-qphwl"
Jul 06 20:25:38.879631 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:38.879605    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b33528c3-08cc-4595-aa46-8a72187a353d-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-qphwl\" (UID: \"b33528c3-08cc-4595-aa46-8a72187a353d\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-qphwl"
Jul 06 20:25:38.887178 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:38.887156    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zm48k\" (UniqueName: \"kubernetes.io/projected/b33528c3-08cc-4595-aa46-8a72187a353d-kube-api-access-zm48k\") pod \"prometheus-operator-567f7b8b7c-qphwl\" (UID: \"b33528c3-08cc-4595-aa46-8a72187a353d\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-qphwl"
Jul 06 20:25:39.234270 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:39.234242    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-tc5ls"
Jul 06 20:25:39.382769 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:39.382715    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/b33528c3-08cc-4595-aa46-8a72187a353d-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-qphwl\" (UID: \"b33528c3-08cc-4595-aa46-8a72187a353d\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-qphwl"
Jul 06 20:25:39.383185 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:25:39.382892    2563 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Jul 06 20:25:39.383185 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:25:39.382971    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b33528c3-08cc-4595-aa46-8a72187a353d-prometheus-operator-tls podName:b33528c3-08cc-4595-aa46-8a72187a353d nodeName:}" failed. No retries permitted until 2026-07-06 20:25:40.382950842 +0000 UTC m=+104.099061493 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/b33528c3-08cc-4595-aa46-8a72187a353d-prometheus-operator-tls") pod "prometheus-operator-567f7b8b7c-qphwl" (UID: "b33528c3-08cc-4595-aa46-8a72187a353d") : secret "prometheus-operator-tls" not found
Jul 06 20:25:39.542567 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:39.542477    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:25:39.929046 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:39.929007    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:25:40.352856 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:40.352821    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:25:40.392129 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:40.392101    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/b33528c3-08cc-4595-aa46-8a72187a353d-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-qphwl\" (UID: \"b33528c3-08cc-4595-aa46-8a72187a353d\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-qphwl"
Jul 06 20:25:40.394403 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:40.394375    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/b33528c3-08cc-4595-aa46-8a72187a353d-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-qphwl\" (UID: \"b33528c3-08cc-4595-aa46-8a72187a353d\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-qphwl"
Jul 06 20:25:40.545367 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:40.545324    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-qphwl"
Jul 06 20:25:40.800480 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:40.800457    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-qphwl"]
Jul 06 20:25:40.803317 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:25:40.803293    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb33528c3_08cc_4595_aa46_8a72187a353d.slice/crio-6101228028d1f0702a03a253f7bfbc9ddf3ce76d65626eaebb5b772fa9d7dec3 WatchSource:0}: Error finding container 6101228028d1f0702a03a253f7bfbc9ddf3ce76d65626eaebb5b772fa9d7dec3: Status 404 returned error can't find the container with id 6101228028d1f0702a03a253f7bfbc9ddf3ce76d65626eaebb5b772fa9d7dec3
Jul 06 20:25:41.349087 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:41.349052    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-qphwl" event={"ID":"b33528c3-08cc-4595-aa46-8a72187a353d","Type":"ContainerStarted","Data":"6101228028d1f0702a03a253f7bfbc9ddf3ce76d65626eaebb5b772fa9d7dec3"}
Jul 06 20:25:41.351012 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:41.350988    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hg45g" event={"ID":"abbed80c-bca8-49ce-91e5-692152f3d0e5","Type":"ContainerStarted","Data":"f64c8b3fc9449dbde98152d8f2e25821d19424475a1364a12839f51eaed2d89f"}
Jul 06 20:25:41.352553 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:41.352534    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gqvcm" event={"ID":"f2c17640-14e9-4b68-93a4-9183cbc2ccd0","Type":"ContainerStarted","Data":"614130de8b3531937f62f7d310b5a07e1b8c6249b19d85acccba22276d33f13a"}
Jul 06 20:25:41.352629 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:41.352572    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gqvcm" event={"ID":"f2c17640-14e9-4b68-93a4-9183cbc2ccd0","Type":"ContainerStarted","Data":"03cef321324e3d4cb10a227114edcb7824c0077c05d7b5cba977e80d454e0fad"}
Jul 06 20:25:41.352684 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:41.352664    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-gqvcm"
Jul 06 20:25:41.370020 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:41.369976    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-hg45g" podStartSLOduration=33.866123783 podStartE2EDuration="41.36996169s" totalImagesPullingTime="7.503837907s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:25:00 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:25:41.369322047 +0000 UTC m=+105.085432702" watchObservedRunningTime="2026-07-06 20:25:41.36996169 +0000 UTC m=+105.086072351"
Jul 06 20:25:41.389562 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:41.387395    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-gqvcm" podStartSLOduration=65.23025774 podStartE2EDuration="1m8.387377287s" totalImagesPullingTime="3.157119547s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:24:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:25:41.385840277 +0000 UTC m=+105.101950939" watchObservedRunningTime="2026-07-06 20:25:41.387377287 +0000 UTC m=+105.103487950"
Jul 06 20:25:42.967479 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:42.967450    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:25:43.060756 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:43.060716    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:25:43.181311 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:43.181224    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:25:43.361287 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:43.361253    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-qphwl" event={"ID":"b33528c3-08cc-4595-aa46-8a72187a353d","Type":"ContainerStarted","Data":"b7749158e93f63a4259ae9c46b79fceb820c84d159df19a640fad62093394c08"}
Jul 06 20:25:43.361461 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:43.361302    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-qphwl" event={"ID":"b33528c3-08cc-4595-aa46-8a72187a353d","Type":"ContainerStarted","Data":"8887926a6cdc3f457b523771833322bc3578153dcafde36148dea8bd46717ea9"}
Jul 06 20:25:43.379503 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:43.379459    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-qphwl" podStartSLOduration=3.343550237 podStartE2EDuration="5.379443362s" totalImagesPullingTime="2.035893125s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:25:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:25:43.378462684 +0000 UTC m=+107.094573346" watchObservedRunningTime="2026-07-06 20:25:43.379443362 +0000 UTC m=+107.095554048"
Jul 06 20:25:43.954573 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:43.954540    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:25:44.873205 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:44.873177    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:25:44.963932 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:44.963892    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:25:45.074464 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:45.074431    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:25:45.129382 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:45.129312    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-88wl5"]
Jul 06 20:25:45.132258 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:45.132235    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-88wl5"
Jul 06 20:25:45.135465 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:45.135446    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Jul 06 20:25:45.135465 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:45.135456    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Jul 06 20:25:45.135627 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:45.135452    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Jul 06 20:25:45.135812 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:45.135798    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-x4f7n\""
Jul 06 20:25:45.143100 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:45.143074    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-kc45l"]
Jul 06 20:25:45.144967 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:45.144950    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-kc45l"
Jul 06 20:25:45.149473 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:45.149452    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-h2cwn\""
Jul 06 20:25:45.149761 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:45.149724    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 06 20:25:45.149965 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:45.149756    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 06 20:25:45.150659 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:45.150642    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 06 20:25:45.156858 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:45.156838    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-88wl5"]
Jul 06 20:25:45.188076 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:45.188052    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:25:45.233171 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:45.233140    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2640cc2d-7e6f-4eb0-a176-eb62a152789a-node-exporter-wtmp\") pod \"node-exporter-kc45l\" (UID: \"2640cc2d-7e6f-4eb0-a176-eb62a152789a\") " pod="openshift-monitoring/node-exporter-kc45l"
Jul 06 20:25:45.233171 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:45.233166    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2640cc2d-7e6f-4eb0-a176-eb62a152789a-root\") pod \"node-exporter-kc45l\" (UID: \"2640cc2d-7e6f-4eb0-a176-eb62a152789a\") " pod="openshift-monitoring/node-exporter-kc45l"
Jul 06 20:25:45.233351 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:45.233189    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2640cc2d-7e6f-4eb0-a176-eb62a152789a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kc45l\" (UID: \"2640cc2d-7e6f-4eb0-a176-eb62a152789a\") " pod="openshift-monitoring/node-exporter-kc45l"
Jul 06 20:25:45.233351 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:45.233207    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w66fd\" (UniqueName: \"kubernetes.io/projected/2640cc2d-7e6f-4eb0-a176-eb62a152789a-kube-api-access-w66fd\") pod \"node-exporter-kc45l\" (UID: \"2640cc2d-7e6f-4eb0-a176-eb62a152789a\") " pod="openshift-monitoring/node-exporter-kc45l"
Jul 06 20:25:45.233351 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:45.233296    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2640cc2d-7e6f-4eb0-a176-eb62a152789a-metrics-client-ca\") pod \"node-exporter-kc45l\" (UID: \"2640cc2d-7e6f-4eb0-a176-eb62a152789a\") " pod="openshift-monitoring/node-exporter-kc45l"
Jul 06 20:25:45.233351 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:45.233327    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/d10133bf-b0d1-4ce9-9906-59fbafad0cc5-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-88wl5\" (UID: \"d10133bf-b0d1-4ce9-9906-59fbafad0cc5\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-88wl5"
Jul 06 20:25:45.233553 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:45.233355    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2640cc2d-7e6f-4eb0-a176-eb62a152789a-sys\") pod \"node-exporter-kc45l\" (UID: \"2640cc2d-7e6f-4eb0-a176-eb62a152789a\") " pod="openshift-monitoring/node-exporter-kc45l"
Jul 06 20:25:45.233553 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:45.233382    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2640cc2d-7e6f-4eb0-a176-eb62a152789a-node-exporter-tls\") pod \"node-exporter-kc45l\" (UID: \"2640cc2d-7e6f-4eb0-a176-eb62a152789a\") " pod="openshift-monitoring/node-exporter-kc45l"
Jul 06 20:25:45.233553 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:45.233407    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d10133bf-b0d1-4ce9-9906-59fbafad0cc5-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-88wl5\" (UID: \"d10133bf-b0d1-4ce9-9906-59fbafad0cc5\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-88wl5"
Jul 06 20:25:45.233553 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:45.233441    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2640cc2d-7e6f-4eb0-a176-eb62a152789a-node-exporter-accelerators-collector-config\") pod \"node-exporter-kc45l\" (UID: \"2640cc2d-7e6f-4eb0-a176-eb62a152789a\") " pod="openshift-monitoring/node-exporter-kc45l"
Jul 06 20:25:45.233553 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:45.233500    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2640cc2d-7e6f-4eb0-a176-eb62a152789a-node-exporter-textfile\") pod \"node-exporter-kc45l\" (UID: \"2640cc2d-7e6f-4eb0-a176-eb62a152789a\") " pod="openshift-monitoring/node-exporter-kc45l"
Jul 06 20:25:45.233806 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:45.233552    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d10133bf-b0d1-4ce9-9906-59fbafad0cc5-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-88wl5\" (UID: \"d10133bf-b0d1-4ce9-9906-59fbafad0cc5\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-88wl5"
Jul 06 20:25:45.233806 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:45.233591    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/d10133bf-b0d1-4ce9-9906-59fbafad0cc5-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-88wl5\" (UID: \"d10133bf-b0d1-4ce9-9906-59fbafad0cc5\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-88wl5"
Jul 06 20:25:45.233806 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:45.233655    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmnfc\" (UniqueName: \"kubernetes.io/projected/d10133bf-b0d1-4ce9-9906-59fbafad0cc5-kube-api-access-lmnfc\") pod \"kube-state-metrics-7c9dbb8b7d-88wl5\" (UID: \"d10133bf-b0d1-4ce9-9906-59fbafad0cc5\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-88wl5"
Jul 06 20:25:45.233806 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:45.233683    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d10133bf-b0d1-4ce9-9906-59fbafad0cc5-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-88wl5\" (UID: \"d10133bf-b0d1-4ce9-9906-59fbafad0cc5\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-88wl5"
Jul 06 20:25:45.334284 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:45.334253    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d10133bf-b0d1-4ce9-9906-59fbafad0cc5-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-88wl5\" (UID: \"d10133bf-b0d1-4ce9-9906-59fbafad0cc5\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-88wl5"
Jul 06 20:25:45.334450 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:45.334294    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2640cc2d-7e6f-4eb0-a176-eb62a152789a-node-exporter-wtmp\") pod \"node-exporter-kc45l\" (UID: \"2640cc2d-7e6f-4eb0-a176-eb62a152789a\") " pod="openshift-monitoring/node-exporter-kc45l"
Jul 06 20:25:45.334450 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:45.334322    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2640cc2d-7e6f-4eb0-a176-eb62a152789a-root\") pod \"node-exporter-kc45l\" (UID: \"2640cc2d-7e6f-4eb0-a176-eb62a152789a\") " pod="openshift-monitoring/node-exporter-kc45l"
Jul 06 20:25:45.334450 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:45.334386    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2640cc2d-7e6f-4eb0-a176-eb62a152789a-root\") pod \"node-exporter-kc45l\" (UID: \"2640cc2d-7e6f-4eb0-a176-eb62a152789a\") " pod="openshift-monitoring/node-exporter-kc45l"
Jul 06 20:25:45.334450 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:45.334421    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2640cc2d-7e6f-4eb0-a176-eb62a152789a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kc45l\" (UID: \"2640cc2d-7e6f-4eb0-a176-eb62a152789a\") " pod="openshift-monitoring/node-exporter-kc45l"
Jul 06 20:25:45.334450 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:45.334447    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w66fd\" (UniqueName: \"kubernetes.io/projected/2640cc2d-7e6f-4eb0-a176-eb62a152789a-kube-api-access-w66fd\") pod \"node-exporter-kc45l\" (UID: \"2640cc2d-7e6f-4eb0-a176-eb62a152789a\") " pod="openshift-monitoring/node-exporter-kc45l"
Jul 06 20:25:45.334709 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:45.334458    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2640cc2d-7e6f-4eb0-a176-eb62a152789a-node-exporter-wtmp\") pod \"node-exporter-kc45l\" (UID: \"2640cc2d-7e6f-4eb0-a176-eb62a152789a\") " pod="openshift-monitoring/node-exporter-kc45l"
Jul 06 20:25:45.334709 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:45.334480    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2640cc2d-7e6f-4eb0-a176-eb62a152789a-metrics-client-ca\") pod \"node-exporter-kc45l\" (UID: \"2640cc2d-7e6f-4eb0-a176-eb62a152789a\") " pod="openshift-monitoring/node-exporter-kc45l"
Jul 06 20:25:45.334709 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:45.334523    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/d10133bf-b0d1-4ce9-9906-59fbafad0cc5-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-88wl5\" (UID: \"d10133bf-b0d1-4ce9-9906-59fbafad0cc5\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-88wl5"
Jul 06 20:25:45.334709 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:45.334548    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2640cc2d-7e6f-4eb0-a176-eb62a152789a-sys\") pod \"node-exporter-kc45l\" (UID: \"2640cc2d-7e6f-4eb0-a176-eb62a152789a\") " pod="openshift-monitoring/node-exporter-kc45l"
Jul 06 20:25:45.334709 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:45.334574    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2640cc2d-7e6f-4eb0-a176-eb62a152789a-node-exporter-tls\") pod \"node-exporter-kc45l\" (UID: \"2640cc2d-7e6f-4eb0-a176-eb62a152789a\") " pod="openshift-monitoring/node-exporter-kc45l"
Jul 06 20:25:45.334709 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:45.334601    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d10133bf-b0d1-4ce9-9906-59fbafad0cc5-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-88wl5\" (UID: \"d10133bf-b0d1-4ce9-9906-59fbafad0cc5\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-88wl5"
Jul 06 20:25:45.334709 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:45.334649    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2640cc2d-7e6f-4eb0-a176-eb62a152789a-node-exporter-accelerators-collector-config\") pod \"node-exporter-kc45l\" (UID: \"2640cc2d-7e6f-4eb0-a176-eb62a152789a\") " pod="openshift-monitoring/node-exporter-kc45l"
Jul 06 20:25:45.334709 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:45.334683    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2640cc2d-7e6f-4eb0-a176-eb62a152789a-node-exporter-textfile\") pod \"node-exporter-kc45l\" (UID: \"2640cc2d-7e6f-4eb0-a176-eb62a152789a\") " pod="openshift-monitoring/node-exporter-kc45l"
Jul 06 20:25:45.335095 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:45.334749    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d10133bf-b0d1-4ce9-9906-59fbafad0cc5-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-88wl5\" (UID: \"d10133bf-b0d1-4ce9-9906-59fbafad0cc5\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-88wl5"
Jul 06 20:25:45.335095 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:45.334791    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2640cc2d-7e6f-4eb0-a176-eb62a152789a-sys\") pod \"node-exporter-kc45l\" (UID: \"2640cc2d-7e6f-4eb0-a176-eb62a152789a\") " pod="openshift-monitoring/node-exporter-kc45l"
Jul 06 20:25:45.335095 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:45.334793    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/d10133bf-b0d1-4ce9-9906-59fbafad0cc5-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-88wl5\" (UID: \"d10133bf-b0d1-4ce9-9906-59fbafad0cc5\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-88wl5"
Jul 06 20:25:45.335095 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:45.334849    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmnfc\" (UniqueName: \"kubernetes.io/projected/d10133bf-b0d1-4ce9-9906-59fbafad0cc5-kube-api-access-lmnfc\") pod \"kube-state-metrics-7c9dbb8b7d-88wl5\" (UID: \"d10133bf-b0d1-4ce9-9906-59fbafad0cc5\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-88wl5"
Jul 06 20:25:45.335546 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:45.335513    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/d10133bf-b0d1-4ce9-9906-59fbafad0cc5-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-88wl5\" (UID: \"d10133bf-b0d1-4ce9-9906-59fbafad0cc5\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-88wl5"
Jul 06 20:25:45.338034 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:45.338007    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2640cc2d-7e6f-4eb0-a176-eb62a152789a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kc45l\" (UID: \"2640cc2d-7e6f-4eb0-a176-eb62a152789a\") " pod="openshift-monitoring/node-exporter-kc45l"
Jul 06 20:25:45.347123 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:45.335423    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/d10133bf-b0d1-4ce9-9906-59fbafad0cc5-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-88wl5\" (UID: \"d10133bf-b0d1-4ce9-9906-59fbafad0cc5\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-88wl5"
Jul 06 20:25:45.347123 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:25:45.338876    2563 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Jul 06 20:25:45.347123 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:45.338895    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2640cc2d-7e6f-4eb0-a176-eb62a152789a-node-exporter-accelerators-collector-config\") pod \"node-exporter-kc45l\" (UID: \"2640cc2d-7e6f-4eb0-a176-eb62a152789a\") " pod="openshift-monitoring/node-exporter-kc45l"
Jul 06 20:25:45.347123 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:25:45.338950    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d10133bf-b0d1-4ce9-9906-59fbafad0cc5-kube-state-metrics-tls podName:d10133bf-b0d1-4ce9-9906-59fbafad0cc5 nodeName:}" failed. No retries permitted until 2026-07-06 20:25:45.838929705 +0000 UTC m=+109.555040358 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/d10133bf-b0d1-4ce9-9906-59fbafad0cc5-kube-state-metrics-tls") pod "kube-state-metrics-7c9dbb8b7d-88wl5" (UID: "d10133bf-b0d1-4ce9-9906-59fbafad0cc5") : secret "kube-state-metrics-tls" not found
Jul 06 20:25:45.347123 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:45.339809    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d10133bf-b0d1-4ce9-9906-59fbafad0cc5-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-88wl5\" (UID: \"d10133bf-b0d1-4ce9-9906-59fbafad0cc5\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-88wl5"
Jul 06 20:25:45.347123 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:25:45.339917    2563 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 06 20:25:45.347123 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:25:45.339976    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2640cc2d-7e6f-4eb0-a176-eb62a152789a-node-exporter-tls podName:2640cc2d-7e6f-4eb0-a176-eb62a152789a nodeName:}" failed. No retries permitted until 2026-07-06 20:25:45.839958033 +0000 UTC m=+109.556068687 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/2640cc2d-7e6f-4eb0-a176-eb62a152789a-node-exporter-tls") pod "node-exporter-kc45l" (UID: "2640cc2d-7e6f-4eb0-a176-eb62a152789a") : secret "node-exporter-tls" not found
Jul 06 20:25:45.347123 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:45.340011    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2640cc2d-7e6f-4eb0-a176-eb62a152789a-node-exporter-textfile\") pod \"node-exporter-kc45l\" (UID: \"2640cc2d-7e6f-4eb0-a176-eb62a152789a\") " pod="openshift-monitoring/node-exporter-kc45l"
Jul 06 20:25:45.347123 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:45.340159    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d10133bf-b0d1-4ce9-9906-59fbafad0cc5-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-88wl5\" (UID: \"d10133bf-b0d1-4ce9-9906-59fbafad0cc5\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-88wl5"
Jul 06 20:25:45.347123 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:45.344420    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmnfc\" (UniqueName: \"kubernetes.io/projected/d10133bf-b0d1-4ce9-9906-59fbafad0cc5-kube-api-access-lmnfc\") pod \"kube-state-metrics-7c9dbb8b7d-88wl5\" (UID: \"d10133bf-b0d1-4ce9-9906-59fbafad0cc5\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-88wl5"
Jul 06 20:25:45.347678 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:45.335400    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2640cc2d-7e6f-4eb0-a176-eb62a152789a-metrics-client-ca\") pod \"node-exporter-kc45l\" (UID: \"2640cc2d-7e6f-4eb0-a176-eb62a152789a\") " pod="openshift-monitoring/node-exporter-kc45l"
Jul 06 20:25:45.350187 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:45.350145    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w66fd\" (UniqueName: \"kubernetes.io/projected/2640cc2d-7e6f-4eb0-a176-eb62a152789a-kube-api-access-w66fd\") pod \"node-exporter-kc45l\" (UID: \"2640cc2d-7e6f-4eb0-a176-eb62a152789a\") " pod="openshift-monitoring/node-exporter-kc45l"
Jul 06 20:25:45.370075 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:45.370046    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:25:45.474918 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:45.474829    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:25:45.577277 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:45.577239    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:25:45.671239 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:45.671200    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:25:45.784587 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:45.784497    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:25:45.839697 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:45.839666    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d10133bf-b0d1-4ce9-9906-59fbafad0cc5-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-88wl5\" (UID: \"d10133bf-b0d1-4ce9-9906-59fbafad0cc5\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-88wl5"
Jul 06 20:25:45.842067 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:45.842043    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d10133bf-b0d1-4ce9-9906-59fbafad0cc5-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-88wl5\" (UID: \"d10133bf-b0d1-4ce9-9906-59fbafad0cc5\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-88wl5"
Jul 06 20:25:45.940422 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:45.940389    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2640cc2d-7e6f-4eb0-a176-eb62a152789a-node-exporter-tls\") pod \"node-exporter-kc45l\" (UID: \"2640cc2d-7e6f-4eb0-a176-eb62a152789a\") " pod="openshift-monitoring/node-exporter-kc45l"
Jul 06 20:25:45.942642 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:45.942625    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2640cc2d-7e6f-4eb0-a176-eb62a152789a-node-exporter-tls\") pod \"node-exporter-kc45l\" (UID: \"2640cc2d-7e6f-4eb0-a176-eb62a152789a\") " pod="openshift-monitoring/node-exporter-kc45l"
Jul 06 20:25:46.043952 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:46.043893    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-88wl5"
Jul 06 20:25:46.053314 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:46.053291    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-kc45l"
Jul 06 20:25:46.061827 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:25:46.061800    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2640cc2d_7e6f_4eb0_a176_eb62a152789a.slice/crio-5db8113ad6bea73230e87cc9309c5447cf048fc09d5c31a79ac1463f84d22f78 WatchSource:0}: Error finding container 5db8113ad6bea73230e87cc9309c5447cf048fc09d5c31a79ac1463f84d22f78: Status 404 returned error can't find the container with id 5db8113ad6bea73230e87cc9309c5447cf048fc09d5c31a79ac1463f84d22f78
Jul 06 20:25:46.132951 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:46.132919    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:25:46.186995 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:46.186912    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-88wl5"]
Jul 06 20:25:46.189380 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:25:46.189350    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd10133bf_b0d1_4ce9_9906_59fbafad0cc5.slice/crio-bde2a1342ae6efc9c657bac16ed903b3a4507828f9537d2a817584c064615340 WatchSource:0}: Error finding container bde2a1342ae6efc9c657bac16ed903b3a4507828f9537d2a817584c064615340: Status 404 returned error can't find the container with id bde2a1342ae6efc9c657bac16ed903b3a4507828f9537d2a817584c064615340
Jul 06 20:25:46.237443 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:46.237406    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:25:46.247667 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:46.247641    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 06 20:25:46.251006 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:46.250992    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:25:46.260445 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:46.260422    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 06 20:25:46.261366 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:46.261351    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 06 20:25:46.266614 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:46.266601    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 06 20:25:46.277449 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:46.277431    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 06 20:25:46.282053 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:46.282030    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 06 20:25:46.294182 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:46.294124    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 06 20:25:46.294182 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:46.294124    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 06 20:25:46.306544 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:46.306524    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 06 20:25:46.306621 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:46.306542    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 06 20:25:46.306621 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:46.306558    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 06 20:25:46.311382 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:46.311368    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-t49cg\""
Jul 06 20:25:46.343687 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:46.343663    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f7dadb57-9f0f-4e2d-9a22-0d9e200528e1-config-volume\") pod \"alertmanager-main-0\" (UID: \"f7dadb57-9f0f-4e2d-9a22-0d9e200528e1\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:25:46.343785 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:46.343708    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f7dadb57-9f0f-4e2d-9a22-0d9e200528e1-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f7dadb57-9f0f-4e2d-9a22-0d9e200528e1\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:25:46.343839 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:46.343805    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f7dadb57-9f0f-4e2d-9a22-0d9e200528e1-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f7dadb57-9f0f-4e2d-9a22-0d9e200528e1\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:25:46.343839 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:46.343830    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f7dadb57-9f0f-4e2d-9a22-0d9e200528e1-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f7dadb57-9f0f-4e2d-9a22-0d9e200528e1\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:25:46.343929 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:46.343912    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f7dadb57-9f0f-4e2d-9a22-0d9e200528e1-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f7dadb57-9f0f-4e2d-9a22-0d9e200528e1\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:25:46.343964 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:46.343952    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f7dadb57-9f0f-4e2d-9a22-0d9e200528e1-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f7dadb57-9f0f-4e2d-9a22-0d9e200528e1\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:25:46.344012 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:46.343999    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f7dadb57-9f0f-4e2d-9a22-0d9e200528e1-web-config\") pod \"alertmanager-main-0\" (UID: \"f7dadb57-9f0f-4e2d-9a22-0d9e200528e1\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:25:46.344048 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:46.344021    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f7dadb57-9f0f-4e2d-9a22-0d9e200528e1-config-out\") pod \"alertmanager-main-0\" (UID: \"f7dadb57-9f0f-4e2d-9a22-0d9e200528e1\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:25:46.344048 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:46.344040    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f7dadb57-9f0f-4e2d-9a22-0d9e200528e1-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f7dadb57-9f0f-4e2d-9a22-0d9e200528e1\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:25:46.344121 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:46.344070    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvm2k\" (UniqueName: \"kubernetes.io/projected/f7dadb57-9f0f-4e2d-9a22-0d9e200528e1-kube-api-access-hvm2k\") pod \"alertmanager-main-0\" (UID: \"f7dadb57-9f0f-4e2d-9a22-0d9e200528e1\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:25:46.344152 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:46.344122    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f7dadb57-9f0f-4e2d-9a22-0d9e200528e1-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f7dadb57-9f0f-4e2d-9a22-0d9e200528e1\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:25:46.344181 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:46.344156    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7dadb57-9f0f-4e2d-9a22-0d9e200528e1-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f7dadb57-9f0f-4e2d-9a22-0d9e200528e1\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:25:46.344225 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:46.344208    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f7dadb57-9f0f-4e2d-9a22-0d9e200528e1-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f7dadb57-9f0f-4e2d-9a22-0d9e200528e1\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:25:46.372849 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:46.372818    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kc45l" event={"ID":"2640cc2d-7e6f-4eb0-a176-eb62a152789a","Type":"ContainerStarted","Data":"5db8113ad6bea73230e87cc9309c5447cf048fc09d5c31a79ac1463f84d22f78"}
Jul 06 20:25:46.374025 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:46.374006    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-88wl5" event={"ID":"d10133bf-b0d1-4ce9-9906-59fbafad0cc5","Type":"ContainerStarted","Data":"bde2a1342ae6efc9c657bac16ed903b3a4507828f9537d2a817584c064615340"}
Jul 06 20:25:46.431184 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:46.431158    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:25:46.445250 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:46.445224    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f7dadb57-9f0f-4e2d-9a22-0d9e200528e1-config-volume\") pod \"alertmanager-main-0\" (UID: \"f7dadb57-9f0f-4e2d-9a22-0d9e200528e1\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:25:46.445366 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:46.445257    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f7dadb57-9f0f-4e2d-9a22-0d9e200528e1-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f7dadb57-9f0f-4e2d-9a22-0d9e200528e1\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:25:46.445366 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:46.445305    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f7dadb57-9f0f-4e2d-9a22-0d9e200528e1-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f7dadb57-9f0f-4e2d-9a22-0d9e200528e1\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:25:46.445366 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:46.445329    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f7dadb57-9f0f-4e2d-9a22-0d9e200528e1-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f7dadb57-9f0f-4e2d-9a22-0d9e200528e1\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:25:46.445366 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:46.445363    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f7dadb57-9f0f-4e2d-9a22-0d9e200528e1-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f7dadb57-9f0f-4e2d-9a22-0d9e200528e1\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:25:46.445550 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:46.445404    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f7dadb57-9f0f-4e2d-9a22-0d9e200528e1-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f7dadb57-9f0f-4e2d-9a22-0d9e200528e1\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:25:46.445550 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:46.445520    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f7dadb57-9f0f-4e2d-9a22-0d9e200528e1-web-config\") pod \"alertmanager-main-0\" (UID: \"f7dadb57-9f0f-4e2d-9a22-0d9e200528e1\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:25:46.445633 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:46.445563    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f7dadb57-9f0f-4e2d-9a22-0d9e200528e1-config-out\") pod \"alertmanager-main-0\" (UID: \"f7dadb57-9f0f-4e2d-9a22-0d9e200528e1\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:25:46.445675 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:46.445624    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f7dadb57-9f0f-4e2d-9a22-0d9e200528e1-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f7dadb57-9f0f-4e2d-9a22-0d9e200528e1\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:25:46.445715 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:46.445672    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvm2k\" (UniqueName: \"kubernetes.io/projected/f7dadb57-9f0f-4e2d-9a22-0d9e200528e1-kube-api-access-hvm2k\") pod \"alertmanager-main-0\" (UID: \"f7dadb57-9f0f-4e2d-9a22-0d9e200528e1\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:25:46.445779 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:46.445720    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f7dadb57-9f0f-4e2d-9a22-0d9e200528e1-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f7dadb57-9f0f-4e2d-9a22-0d9e200528e1\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:25:46.445829 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:46.445774    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7dadb57-9f0f-4e2d-9a22-0d9e200528e1-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f7dadb57-9f0f-4e2d-9a22-0d9e200528e1\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:25:46.445829 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:46.445809    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f7dadb57-9f0f-4e2d-9a22-0d9e200528e1-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f7dadb57-9f0f-4e2d-9a22-0d9e200528e1\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:25:46.446556 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:46.446530    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f7dadb57-9f0f-4e2d-9a22-0d9e200528e1-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f7dadb57-9f0f-4e2d-9a22-0d9e200528e1\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:25:46.446745 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:25:46.446662    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f7dadb57-9f0f-4e2d-9a22-0d9e200528e1-alertmanager-trusted-ca-bundle podName:f7dadb57-9f0f-4e2d-9a22-0d9e200528e1 nodeName:}" failed. No retries permitted until 2026-07-06 20:25:46.94664031 +0000 UTC m=+110.662750963 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/f7dadb57-9f0f-4e2d-9a22-0d9e200528e1-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "f7dadb57-9f0f-4e2d-9a22-0d9e200528e1") : configmap references non-existent config key: ca-bundle.crt
Jul 06 20:25:46.447068 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:46.447044    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f7dadb57-9f0f-4e2d-9a22-0d9e200528e1-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f7dadb57-9f0f-4e2d-9a22-0d9e200528e1\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:25:46.448721 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:46.448689    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f7dadb57-9f0f-4e2d-9a22-0d9e200528e1-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f7dadb57-9f0f-4e2d-9a22-0d9e200528e1\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:25:46.449101 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:46.449081    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f7dadb57-9f0f-4e2d-9a22-0d9e200528e1-config-volume\") pod \"alertmanager-main-0\" (UID: \"f7dadb57-9f0f-4e2d-9a22-0d9e200528e1\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:25:46.449388 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:46.449364    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f7dadb57-9f0f-4e2d-9a22-0d9e200528e1-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f7dadb57-9f0f-4e2d-9a22-0d9e200528e1\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:25:46.449445 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:46.449377    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f7dadb57-9f0f-4e2d-9a22-0d9e200528e1-web-config\") pod \"alertmanager-main-0\" (UID: \"f7dadb57-9f0f-4e2d-9a22-0d9e200528e1\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:25:46.449554 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:46.449533    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f7dadb57-9f0f-4e2d-9a22-0d9e200528e1-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f7dadb57-9f0f-4e2d-9a22-0d9e200528e1\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:25:46.449987 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:46.449967    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f7dadb57-9f0f-4e2d-9a22-0d9e200528e1-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f7dadb57-9f0f-4e2d-9a22-0d9e200528e1\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:25:46.450457 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:46.450429    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f7dadb57-9f0f-4e2d-9a22-0d9e200528e1-config-out\") pod \"alertmanager-main-0\" (UID: \"f7dadb57-9f0f-4e2d-9a22-0d9e200528e1\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:25:46.450758 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:46.450719    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f7dadb57-9f0f-4e2d-9a22-0d9e200528e1-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f7dadb57-9f0f-4e2d-9a22-0d9e200528e1\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:25:46.451296 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:46.451276    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f7dadb57-9f0f-4e2d-9a22-0d9e200528e1-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f7dadb57-9f0f-4e2d-9a22-0d9e200528e1\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:25:46.458330 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:46.458311    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvm2k\" (UniqueName: \"kubernetes.io/projected/f7dadb57-9f0f-4e2d-9a22-0d9e200528e1-kube-api-access-hvm2k\") pod \"alertmanager-main-0\" (UID: \"f7dadb57-9f0f-4e2d-9a22-0d9e200528e1\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:25:46.532105 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:46.532062    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:25:46.950505 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:46.950473    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7dadb57-9f0f-4e2d-9a22-0d9e200528e1-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f7dadb57-9f0f-4e2d-9a22-0d9e200528e1\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:25:46.951789 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:46.951740    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7dadb57-9f0f-4e2d-9a22-0d9e200528e1-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f7dadb57-9f0f-4e2d-9a22-0d9e200528e1\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:25:47.160987 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:47.160953    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:25:47.773629 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:47.773496    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 06 20:25:47.811043 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:25:47.811007    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7dadb57_9f0f_4e2d_9a22_0d9e200528e1.slice/crio-837c9068e192a0ef36f6dc32a898f081f20d53dd3e9518a76b80e5601be31d1f WatchSource:0}: Error finding container 837c9068e192a0ef36f6dc32a898f081f20d53dd3e9518a76b80e5601be31d1f: Status 404 returned error can't find the container with id 837c9068e192a0ef36f6dc32a898f081f20d53dd3e9518a76b80e5601be31d1f
Jul 06 20:25:48.139086 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:48.139022    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:25:48.240295 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:48.240262    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:25:48.315705 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:48.315665    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:25:48.382817 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:48.382785    2563 generic.go:356] "Generic (PLEG): container finished" podID="2640cc2d-7e6f-4eb0-a176-eb62a152789a" containerID="c376750ea7ecd01769872ace06ac7f385035e5e8d27974a34b7a1c48e47f7b82" exitCode=0
Jul 06 20:25:48.382998 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:48.382881    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kc45l" event={"ID":"2640cc2d-7e6f-4eb0-a176-eb62a152789a","Type":"ContainerDied","Data":"c376750ea7ecd01769872ace06ac7f385035e5e8d27974a34b7a1c48e47f7b82"}
Jul 06 20:25:48.385270 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:48.385244    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-88wl5" event={"ID":"d10133bf-b0d1-4ce9-9906-59fbafad0cc5","Type":"ContainerStarted","Data":"f45d9c9988a91048f81f3c4fab1931eb52a30db55eca72dbfe9a493acf0b8501"}
Jul 06 20:25:48.385403 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:48.385302    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-88wl5" event={"ID":"d10133bf-b0d1-4ce9-9906-59fbafad0cc5","Type":"ContainerStarted","Data":"65e6686331d8a17c27ca772ff46ebed0dfe46406cebf01d2ed1cde878878f5fd"}
Jul 06 20:25:48.385403 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:48.385333    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-88wl5" event={"ID":"d10133bf-b0d1-4ce9-9906-59fbafad0cc5","Type":"ContainerStarted","Data":"8521a335f351dcbff23268993851d0178017f8cb8e3f0aeca390fdbbf564410a"}
Jul 06 20:25:48.386378 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:48.386357    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f7dadb57-9f0f-4e2d-9a22-0d9e200528e1","Type":"ContainerStarted","Data":"837c9068e192a0ef36f6dc32a898f081f20d53dd3e9518a76b80e5601be31d1f"}
Jul 06 20:25:48.426523 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:48.426479    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-88wl5" podStartSLOduration=2.02281039 podStartE2EDuration="3.42646184s" totalImagesPullingTime="1.40365145s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:25:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:25:48.425282512 +0000 UTC m=+112.141393172" watchObservedRunningTime="2026-07-06 20:25:48.42646184 +0000 UTC m=+112.142572499"
Jul 06 20:25:48.489876 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:48.489847    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:25:49.198951 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:49.198919    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:25:49.391618 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:49.391581    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kc45l" event={"ID":"2640cc2d-7e6f-4eb0-a176-eb62a152789a","Type":"ContainerStarted","Data":"b7f8104aef554ac62f3e6508d27dc011c0960f4e34b713ae285101856d2ee5c7"}
Jul 06 20:25:49.391835 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:49.391642    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kc45l" event={"ID":"2640cc2d-7e6f-4eb0-a176-eb62a152789a","Type":"ContainerStarted","Data":"8398f4276c4b0cbcc2001b57fc6e08cd5951b8378e63e0c669604f4e66ab734c"}
Jul 06 20:25:49.393266 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:49.393207    2563 generic.go:356] "Generic (PLEG): container finished" podID="f7dadb57-9f0f-4e2d-9a22-0d9e200528e1" containerID="e20b57a7309d0963ab9a112ac25f4fdf8baec86897f4fd9af4cea690f40f382b" exitCode=0
Jul 06 20:25:49.393407 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:49.393303    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f7dadb57-9f0f-4e2d-9a22-0d9e200528e1","Type":"ContainerDied","Data":"e20b57a7309d0963ab9a112ac25f4fdf8baec86897f4fd9af4cea690f40f382b"}
Jul 06 20:25:49.415270 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:49.415213    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-kc45l" podStartSLOduration=2.796821813 podStartE2EDuration="4.4151949s" totalImagesPullingTime="1.530189328s" totalInitContainerRuntime="88.183759ms" isStatefulPod=true podCreationTimestamp="2026-07-06 20:25:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:25:49.414339088 +0000 UTC m=+113.130449748" watchObservedRunningTime="2026-07-06 20:25:49.4151949 +0000 UTC m=+113.131305570"
Jul 06 20:25:49.432587 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:49.432554    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-5b84f65c46-pb5v8"]
Jul 06 20:25:49.434713 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:49.434693    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5b84f65c46-pb5v8"
Jul 06 20:25:49.437007 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:49.436984    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Jul 06 20:25:49.437114 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:49.436985    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-eu3o035km7enk\""
Jul 06 20:25:49.437114 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:49.437026    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-9fpqt\""
Jul 06 20:25:49.437114 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:49.437031    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 06 20:25:49.437375 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:49.437359    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Jul 06 20:25:49.437452 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:49.437437    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Jul 06 20:25:49.446646 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:49.446625    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5b84f65c46-pb5v8"]
Jul 06 20:25:49.475504 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:49.475467    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/6827cc99-95c6-4a25-8e25-e73e7e3e2d7c-secret-metrics-server-tls\") pod \"metrics-server-5b84f65c46-pb5v8\" (UID: \"6827cc99-95c6-4a25-8e25-e73e7e3e2d7c\") " pod="openshift-monitoring/metrics-server-5b84f65c46-pb5v8"
Jul 06 20:25:49.475504 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:49.475506    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/6827cc99-95c6-4a25-8e25-e73e7e3e2d7c-secret-metrics-server-client-certs\") pod \"metrics-server-5b84f65c46-pb5v8\" (UID: \"6827cc99-95c6-4a25-8e25-e73e7e3e2d7c\") " pod="openshift-monitoring/metrics-server-5b84f65c46-pb5v8"
Jul 06 20:25:49.477062 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:49.476002    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzs75\" (UniqueName: \"kubernetes.io/projected/6827cc99-95c6-4a25-8e25-e73e7e3e2d7c-kube-api-access-lzs75\") pod \"metrics-server-5b84f65c46-pb5v8\" (UID: \"6827cc99-95c6-4a25-8e25-e73e7e3e2d7c\") " pod="openshift-monitoring/metrics-server-5b84f65c46-pb5v8"
Jul 06 20:25:49.477062 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:49.476121    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6827cc99-95c6-4a25-8e25-e73e7e3e2d7c-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5b84f65c46-pb5v8\" (UID: \"6827cc99-95c6-4a25-8e25-e73e7e3e2d7c\") " pod="openshift-monitoring/metrics-server-5b84f65c46-pb5v8"
Jul 06 20:25:49.479561 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:49.478023    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/6827cc99-95c6-4a25-8e25-e73e7e3e2d7c-audit-log\") pod \"metrics-server-5b84f65c46-pb5v8\" (UID: \"6827cc99-95c6-4a25-8e25-e73e7e3e2d7c\") " pod="openshift-monitoring/metrics-server-5b84f65c46-pb5v8"
Jul 06 20:25:49.479561 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:49.478197    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/6827cc99-95c6-4a25-8e25-e73e7e3e2d7c-client-ca-bundle\") pod \"metrics-server-5b84f65c46-pb5v8\" (UID: \"6827cc99-95c6-4a25-8e25-e73e7e3e2d7c\") " pod="openshift-monitoring/metrics-server-5b84f65c46-pb5v8"
Jul 06 20:25:49.479561 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:49.478325    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/6827cc99-95c6-4a25-8e25-e73e7e3e2d7c-metrics-server-audit-profiles\") pod \"metrics-server-5b84f65c46-pb5v8\" (UID: \"6827cc99-95c6-4a25-8e25-e73e7e3e2d7c\") " pod="openshift-monitoring/metrics-server-5b84f65c46-pb5v8"
Jul 06 20:25:49.579011 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:49.578971    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/6827cc99-95c6-4a25-8e25-e73e7e3e2d7c-secret-metrics-server-tls\") pod \"metrics-server-5b84f65c46-pb5v8\" (UID: \"6827cc99-95c6-4a25-8e25-e73e7e3e2d7c\") " pod="openshift-monitoring/metrics-server-5b84f65c46-pb5v8"
Jul 06 20:25:49.579011 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:49.579013    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/6827cc99-95c6-4a25-8e25-e73e7e3e2d7c-secret-metrics-server-client-certs\") pod \"metrics-server-5b84f65c46-pb5v8\" (UID: \"6827cc99-95c6-4a25-8e25-e73e7e3e2d7c\") " pod="openshift-monitoring/metrics-server-5b84f65c46-pb5v8"
Jul 06 20:25:49.579266 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:49.579050    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzs75\" (UniqueName: \"kubernetes.io/projected/6827cc99-95c6-4a25-8e25-e73e7e3e2d7c-kube-api-access-lzs75\") pod \"metrics-server-5b84f65c46-pb5v8\" (UID: \"6827cc99-95c6-4a25-8e25-e73e7e3e2d7c\") " pod="openshift-monitoring/metrics-server-5b84f65c46-pb5v8"
Jul 06 20:25:49.579266 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:49.579077    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6827cc99-95c6-4a25-8e25-e73e7e3e2d7c-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5b84f65c46-pb5v8\" (UID: \"6827cc99-95c6-4a25-8e25-e73e7e3e2d7c\") " pod="openshift-monitoring/metrics-server-5b84f65c46-pb5v8"
Jul 06 20:25:49.579266 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:49.579122    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/6827cc99-95c6-4a25-8e25-e73e7e3e2d7c-audit-log\") pod \"metrics-server-5b84f65c46-pb5v8\" (UID: \"6827cc99-95c6-4a25-8e25-e73e7e3e2d7c\") " pod="openshift-monitoring/metrics-server-5b84f65c46-pb5v8"
Jul 06 20:25:49.579266 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:49.579156    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/6827cc99-95c6-4a25-8e25-e73e7e3e2d7c-client-ca-bundle\") pod \"metrics-server-5b84f65c46-pb5v8\" (UID: \"6827cc99-95c6-4a25-8e25-e73e7e3e2d7c\") " pod="openshift-monitoring/metrics-server-5b84f65c46-pb5v8"
Jul 06 20:25:49.579266 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:49.579180    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/6827cc99-95c6-4a25-8e25-e73e7e3e2d7c-metrics-server-audit-profiles\") pod \"metrics-server-5b84f65c46-pb5v8\" (UID: \"6827cc99-95c6-4a25-8e25-e73e7e3e2d7c\") " pod="openshift-monitoring/metrics-server-5b84f65c46-pb5v8"
Jul 06 20:25:49.579572 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:49.579542    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/6827cc99-95c6-4a25-8e25-e73e7e3e2d7c-audit-log\") pod \"metrics-server-5b84f65c46-pb5v8\" (UID: \"6827cc99-95c6-4a25-8e25-e73e7e3e2d7c\") " pod="openshift-monitoring/metrics-server-5b84f65c46-pb5v8"
Jul 06 20:25:49.579911 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:49.579889    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6827cc99-95c6-4a25-8e25-e73e7e3e2d7c-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5b84f65c46-pb5v8\" (UID: \"6827cc99-95c6-4a25-8e25-e73e7e3e2d7c\") " pod="openshift-monitoring/metrics-server-5b84f65c46-pb5v8"
Jul 06 20:25:49.580118 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:49.580100    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/6827cc99-95c6-4a25-8e25-e73e7e3e2d7c-metrics-server-audit-profiles\") pod \"metrics-server-5b84f65c46-pb5v8\" (UID: \"6827cc99-95c6-4a25-8e25-e73e7e3e2d7c\") " pod="openshift-monitoring/metrics-server-5b84f65c46-pb5v8"
Jul 06 20:25:49.581610 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:49.581587    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/6827cc99-95c6-4a25-8e25-e73e7e3e2d7c-client-ca-bundle\") pod \"metrics-server-5b84f65c46-pb5v8\" (UID: \"6827cc99-95c6-4a25-8e25-e73e7e3e2d7c\") " pod="openshift-monitoring/metrics-server-5b84f65c46-pb5v8"
Jul 06 20:25:49.582005 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:49.581988    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/6827cc99-95c6-4a25-8e25-e73e7e3e2d7c-secret-metrics-server-client-certs\") pod \"metrics-server-5b84f65c46-pb5v8\" (UID: \"6827cc99-95c6-4a25-8e25-e73e7e3e2d7c\") " pod="openshift-monitoring/metrics-server-5b84f65c46-pb5v8"
Jul 06 20:25:49.582055 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:49.582030    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/6827cc99-95c6-4a25-8e25-e73e7e3e2d7c-secret-metrics-server-tls\") pod \"metrics-server-5b84f65c46-pb5v8\" (UID: \"6827cc99-95c6-4a25-8e25-e73e7e3e2d7c\") " pod="openshift-monitoring/metrics-server-5b84f65c46-pb5v8"
Jul 06 20:25:49.586996 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:49.586977    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzs75\" (UniqueName: \"kubernetes.io/projected/6827cc99-95c6-4a25-8e25-e73e7e3e2d7c-kube-api-access-lzs75\") pod \"metrics-server-5b84f65c46-pb5v8\" (UID: \"6827cc99-95c6-4a25-8e25-e73e7e3e2d7c\") " pod="openshift-monitoring/metrics-server-5b84f65c46-pb5v8"
Jul 06 20:25:49.745237 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:49.745147    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5b84f65c46-pb5v8"
Jul 06 20:25:49.886882 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:49.886835    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:25:49.887092 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:49.887059    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5b84f65c46-pb5v8"]
Jul 06 20:25:49.890570 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:25:49.890549    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6827cc99_95c6_4a25_8e25_e73e7e3e2d7c.slice/crio-55e3211790da3246c289c6de17c45ab39e2789beebe4f498ae53270f4e87d277 WatchSource:0}: Error finding container 55e3211790da3246c289c6de17c45ab39e2789beebe4f498ae53270f4e87d277: Status 404 returned error can't find the container with id 55e3211790da3246c289c6de17c45ab39e2789beebe4f498ae53270f4e87d277
Jul 06 20:25:49.901315 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:49.901292    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-m2dqb"]
Jul 06 20:25:49.910084 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:49.910063    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-m2dqb"
Jul 06 20:25:49.912454 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:49.912436    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-jfpsd\""
Jul 06 20:25:49.912572 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:49.912468    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Jul 06 20:25:49.916750 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:49.916660    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-m2dqb"]
Jul 06 20:25:49.954892 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:49.954857    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-78b5766db6-4r7qh"]
Jul 06 20:25:49.957124 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:49.957104    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78b5766db6-4r7qh"
Jul 06 20:25:49.959461 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:49.959431    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 06 20:25:49.959596 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:49.959509    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 06 20:25:49.959679 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:49.959599    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 06 20:25:49.959679 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:49.959608    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-g2hph\""
Jul 06 20:25:49.959961 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:49.959941    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 06 20:25:49.960263 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:49.960232    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 06 20:25:49.960263 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:49.960262    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 06 20:25:49.960424 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:49.960235    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 06 20:25:49.967320 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:49.967301    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 06 20:25:49.970289 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:49.970271    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-78b5766db6-4r7qh"]
Jul 06 20:25:49.982913 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:49.982888    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fp2gd\" (UniqueName: \"kubernetes.io/projected/3be44d9c-34b9-4f83-b328-de358e619205-kube-api-access-fp2gd\") pod \"monitoring-plugin-d5dcff7cf-m2dqb\" (UID: \"3be44d9c-34b9-4f83-b328-de358e619205\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-m2dqb"
Jul 06 20:25:49.983058 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:49.982920    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3be44d9c-34b9-4f83-b328-de358e619205-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-m2dqb\" (UID: \"3be44d9c-34b9-4f83-b328-de358e619205\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-m2dqb"
Jul 06 20:25:49.994220 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:49.994188    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:25:50.083618 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:50.083582    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74019db4-ab3b-4e74-8cd0-a0fa363732d6-trusted-ca-bundle\") pod \"console-78b5766db6-4r7qh\" (UID: \"74019db4-ab3b-4e74-8cd0-a0fa363732d6\") " pod="openshift-console/console-78b5766db6-4r7qh"
Jul 06 20:25:50.083807 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:50.083645    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/74019db4-ab3b-4e74-8cd0-a0fa363732d6-console-config\") pod \"console-78b5766db6-4r7qh\" (UID: \"74019db4-ab3b-4e74-8cd0-a0fa363732d6\") " pod="openshift-console/console-78b5766db6-4r7qh"
Jul 06 20:25:50.083807 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:50.083681    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzqsq\" (UniqueName: \"kubernetes.io/projected/74019db4-ab3b-4e74-8cd0-a0fa363732d6-kube-api-access-tzqsq\") pod \"console-78b5766db6-4r7qh\" (UID: \"74019db4-ab3b-4e74-8cd0-a0fa363732d6\") " pod="openshift-console/console-78b5766db6-4r7qh"
Jul 06 20:25:50.083807 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:50.083718    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/74019db4-ab3b-4e74-8cd0-a0fa363732d6-service-ca\") pod \"console-78b5766db6-4r7qh\" (UID: \"74019db4-ab3b-4e74-8cd0-a0fa363732d6\") " pod="openshift-console/console-78b5766db6-4r7qh"
Jul 06 20:25:50.083807 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:50.083779    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/74019db4-ab3b-4e74-8cd0-a0fa363732d6-oauth-serving-cert\") pod \"console-78b5766db6-4r7qh\" (UID: \"74019db4-ab3b-4e74-8cd0-a0fa363732d6\") " pod="openshift-console/console-78b5766db6-4r7qh"
Jul 06 20:25:50.084038 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:50.083818    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/74019db4-ab3b-4e74-8cd0-a0fa363732d6-console-serving-cert\") pod \"console-78b5766db6-4r7qh\" (UID: \"74019db4-ab3b-4e74-8cd0-a0fa363732d6\") " pod="openshift-console/console-78b5766db6-4r7qh"
Jul 06 20:25:50.084038 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:50.083861    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/74019db4-ab3b-4e74-8cd0-a0fa363732d6-console-oauth-config\") pod \"console-78b5766db6-4r7qh\" (UID: \"74019db4-ab3b-4e74-8cd0-a0fa363732d6\") " pod="openshift-console/console-78b5766db6-4r7qh"
Jul 06 20:25:50.084038 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:50.083903    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fp2gd\" (UniqueName: \"kubernetes.io/projected/3be44d9c-34b9-4f83-b328-de358e619205-kube-api-access-fp2gd\") pod \"monitoring-plugin-d5dcff7cf-m2dqb\" (UID: \"3be44d9c-34b9-4f83-b328-de358e619205\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-m2dqb"
Jul 06 20:25:50.084038 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:50.083933    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3be44d9c-34b9-4f83-b328-de358e619205-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-m2dqb\" (UID: \"3be44d9c-34b9-4f83-b328-de358e619205\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-m2dqb"
Jul 06 20:25:50.084201 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:25:50.084086    2563 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Jul 06 20:25:50.084201 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:25:50.084144    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3be44d9c-34b9-4f83-b328-de358e619205-monitoring-plugin-cert podName:3be44d9c-34b9-4f83-b328-de358e619205 nodeName:}" failed. No retries permitted until 2026-07-06 20:25:50.584123772 +0000 UTC m=+114.300234425 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/3be44d9c-34b9-4f83-b328-de358e619205-monitoring-plugin-cert") pod "monitoring-plugin-d5dcff7cf-m2dqb" (UID: "3be44d9c-34b9-4f83-b328-de358e619205") : secret "monitoring-plugin-cert" not found
Jul 06 20:25:50.093670 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:50.093641    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fp2gd\" (UniqueName: \"kubernetes.io/projected/3be44d9c-34b9-4f83-b328-de358e619205-kube-api-access-fp2gd\") pod \"monitoring-plugin-d5dcff7cf-m2dqb\" (UID: \"3be44d9c-34b9-4f83-b328-de358e619205\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-m2dqb"
Jul 06 20:25:50.184889 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:50.184855    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74019db4-ab3b-4e74-8cd0-a0fa363732d6-trusted-ca-bundle\") pod \"console-78b5766db6-4r7qh\" (UID: \"74019db4-ab3b-4e74-8cd0-a0fa363732d6\") " pod="openshift-console/console-78b5766db6-4r7qh"
Jul 06 20:25:50.185089 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:50.184909    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/74019db4-ab3b-4e74-8cd0-a0fa363732d6-console-config\") pod \"console-78b5766db6-4r7qh\" (UID: \"74019db4-ab3b-4e74-8cd0-a0fa363732d6\") " pod="openshift-console/console-78b5766db6-4r7qh"
Jul 06 20:25:50.185089 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:50.184942    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzqsq\" (UniqueName: \"kubernetes.io/projected/74019db4-ab3b-4e74-8cd0-a0fa363732d6-kube-api-access-tzqsq\") pod \"console-78b5766db6-4r7qh\" (UID: \"74019db4-ab3b-4e74-8cd0-a0fa363732d6\") " pod="openshift-console/console-78b5766db6-4r7qh"
Jul 06 20:25:50.185089 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:50.184976    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/74019db4-ab3b-4e74-8cd0-a0fa363732d6-service-ca\") pod \"console-78b5766db6-4r7qh\" (UID: \"74019db4-ab3b-4e74-8cd0-a0fa363732d6\") " pod="openshift-console/console-78b5766db6-4r7qh"
Jul 06 20:25:50.185089 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:50.184999    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/74019db4-ab3b-4e74-8cd0-a0fa363732d6-oauth-serving-cert\") pod \"console-78b5766db6-4r7qh\" (UID: \"74019db4-ab3b-4e74-8cd0-a0fa363732d6\") " pod="openshift-console/console-78b5766db6-4r7qh"
Jul 06 20:25:50.185089 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:50.185028    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/74019db4-ab3b-4e74-8cd0-a0fa363732d6-console-serving-cert\") pod \"console-78b5766db6-4r7qh\" (UID: \"74019db4-ab3b-4e74-8cd0-a0fa363732d6\") " pod="openshift-console/console-78b5766db6-4r7qh"
Jul 06 20:25:50.185336 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:50.185092    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/74019db4-ab3b-4e74-8cd0-a0fa363732d6-console-oauth-config\") pod \"console-78b5766db6-4r7qh\" (UID: \"74019db4-ab3b-4e74-8cd0-a0fa363732d6\") " pod="openshift-console/console-78b5766db6-4r7qh"
Jul 06 20:25:50.185804 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:50.185777    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/74019db4-ab3b-4e74-8cd0-a0fa363732d6-console-config\") pod \"console-78b5766db6-4r7qh\" (UID: \"74019db4-ab3b-4e74-8cd0-a0fa363732d6\") " pod="openshift-console/console-78b5766db6-4r7qh"
Jul 06 20:25:50.185921 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:50.185860    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/74019db4-ab3b-4e74-8cd0-a0fa363732d6-service-ca\") pod \"console-78b5766db6-4r7qh\" (UID: \"74019db4-ab3b-4e74-8cd0-a0fa363732d6\") " pod="openshift-console/console-78b5766db6-4r7qh"
Jul 06 20:25:50.185980 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:50.185945    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74019db4-ab3b-4e74-8cd0-a0fa363732d6-trusted-ca-bundle\") pod \"console-78b5766db6-4r7qh\" (UID: \"74019db4-ab3b-4e74-8cd0-a0fa363732d6\") " pod="openshift-console/console-78b5766db6-4r7qh"
Jul 06 20:25:50.185980 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:50.185950    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/74019db4-ab3b-4e74-8cd0-a0fa363732d6-oauth-serving-cert\") pod \"console-78b5766db6-4r7qh\" (UID: \"74019db4-ab3b-4e74-8cd0-a0fa363732d6\") " pod="openshift-console/console-78b5766db6-4r7qh"
Jul 06 20:25:50.187560 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:50.187532    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/74019db4-ab3b-4e74-8cd0-a0fa363732d6-console-serving-cert\") pod \"console-78b5766db6-4r7qh\" (UID: \"74019db4-ab3b-4e74-8cd0-a0fa363732d6\") " pod="openshift-console/console-78b5766db6-4r7qh"
Jul 06 20:25:50.187652 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:50.187564    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/74019db4-ab3b-4e74-8cd0-a0fa363732d6-console-oauth-config\") pod \"console-78b5766db6-4r7qh\" (UID: \"74019db4-ab3b-4e74-8cd0-a0fa363732d6\") " pod="openshift-console/console-78b5766db6-4r7qh"
Jul 06 20:25:50.194476 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:50.194454    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzqsq\" (UniqueName: \"kubernetes.io/projected/74019db4-ab3b-4e74-8cd0-a0fa363732d6-kube-api-access-tzqsq\") pod \"console-78b5766db6-4r7qh\" (UID: \"74019db4-ab3b-4e74-8cd0-a0fa363732d6\") " pod="openshift-console/console-78b5766db6-4r7qh"
Jul 06 20:25:50.268265 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:50.268227    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78b5766db6-4r7qh"
Jul 06 20:25:50.399879 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:50.399839    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5b84f65c46-pb5v8" event={"ID":"6827cc99-95c6-4a25-8e25-e73e7e3e2d7c","Type":"ContainerStarted","Data":"55e3211790da3246c289c6de17c45ab39e2789beebe4f498ae53270f4e87d277"}
Jul 06 20:25:50.414765 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:50.414636    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-78b5766db6-4r7qh"]
Jul 06 20:25:50.417222 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:25:50.417192    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74019db4_ab3b_4e74_8cd0_a0fa363732d6.slice/crio-19a4e56ae4678b2a259b656d93e244dbb0c1bd45d2d3d5d6313730970be317d1 WatchSource:0}: Error finding container 19a4e56ae4678b2a259b656d93e244dbb0c1bd45d2d3d5d6313730970be317d1: Status 404 returned error can't find the container with id 19a4e56ae4678b2a259b656d93e244dbb0c1bd45d2d3d5d6313730970be317d1
Jul 06 20:25:50.590086 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:50.589917    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3be44d9c-34b9-4f83-b328-de358e619205-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-m2dqb\" (UID: \"3be44d9c-34b9-4f83-b328-de358e619205\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-m2dqb"
Jul 06 20:25:50.590271 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:25:50.590101    2563 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Jul 06 20:25:50.590271 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:25:50.590181    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3be44d9c-34b9-4f83-b328-de358e619205-monitoring-plugin-cert podName:3be44d9c-34b9-4f83-b328-de358e619205 nodeName:}" failed. No retries permitted until 2026-07-06 20:25:51.590159363 +0000 UTC m=+115.306270007 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/3be44d9c-34b9-4f83-b328-de358e619205-monitoring-plugin-cert") pod "monitoring-plugin-d5dcff7cf-m2dqb" (UID: "3be44d9c-34b9-4f83-b328-de358e619205") : secret "monitoring-plugin-cert" not found
Jul 06 20:25:51.020600 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:51.020551    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:25:51.358912 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:51.358878    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-gqvcm"
Jul 06 20:25:51.414949 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:51.409144    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78b5766db6-4r7qh" event={"ID":"74019db4-ab3b-4e74-8cd0-a0fa363732d6","Type":"ContainerStarted","Data":"19a4e56ae4678b2a259b656d93e244dbb0c1bd45d2d3d5d6313730970be317d1"}
Jul 06 20:25:51.443033 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:51.442970    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:25:51.449998 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:51.449264    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 06 20:25:51.453063 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:51.453027    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:25:51.456377 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:51.456345    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 06 20:25:51.456515 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:51.456348    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 06 20:25:51.456716 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:51.456684    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 06 20:25:51.456920 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:51.456902    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 06 20:25:51.457451 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:51.457432    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-7826s\""
Jul 06 20:25:51.460329 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:51.458309    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 06 20:25:51.460329 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:51.458497    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 06 20:25:51.460329 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:51.458569    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 06 20:25:51.460545 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:51.460487    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 06 20:25:51.460699 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:51.460658    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 06 20:25:51.461082 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:51.460958    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 06 20:25:51.461184 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:51.461104    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 06 20:25:51.461253 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:51.461210    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 06 20:25:51.461560 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:51.461538    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-fdef1ap23nrbg\""
Jul 06 20:25:51.464321 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:51.464301    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 06 20:25:51.464889 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:51.464865    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 06 20:25:51.474046 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:51.474015    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 06 20:25:51.500151 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:51.500117    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9155b9e0-f958-4865-a2a2-6efdaa7e9f2f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"9155b9e0-f958-4865-a2a2-6efdaa7e9f2f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:25:51.500320 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:51.500157    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9155b9e0-f958-4865-a2a2-6efdaa7e9f2f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"9155b9e0-f958-4865-a2a2-6efdaa7e9f2f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:25:51.500320 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:51.500189    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/9155b9e0-f958-4865-a2a2-6efdaa7e9f2f-config\") pod \"prometheus-k8s-0\" (UID: \"9155b9e0-f958-4865-a2a2-6efdaa7e9f2f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:25:51.500320 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:51.500218    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9155b9e0-f958-4865-a2a2-6efdaa7e9f2f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"9155b9e0-f958-4865-a2a2-6efdaa7e9f2f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:25:51.500320 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:51.500253    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9155b9e0-f958-4865-a2a2-6efdaa7e9f2f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"9155b9e0-f958-4865-a2a2-6efdaa7e9f2f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:25:51.500320 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:51.500275    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9155b9e0-f958-4865-a2a2-6efdaa7e9f2f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"9155b9e0-f958-4865-a2a2-6efdaa7e9f2f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:25:51.500713 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:51.500435    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9155b9e0-f958-4865-a2a2-6efdaa7e9f2f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"9155b9e0-f958-4865-a2a2-6efdaa7e9f2f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:25:51.500713 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:51.500461    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9155b9e0-f958-4865-a2a2-6efdaa7e9f2f-config-out\") pod \"prometheus-k8s-0\" (UID: \"9155b9e0-f958-4865-a2a2-6efdaa7e9f2f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:25:51.500713 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:51.500485    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9155b9e0-f958-4865-a2a2-6efdaa7e9f2f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"9155b9e0-f958-4865-a2a2-6efdaa7e9f2f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:25:51.500713 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:51.500517    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9155b9e0-f958-4865-a2a2-6efdaa7e9f2f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"9155b9e0-f958-4865-a2a2-6efdaa7e9f2f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:25:51.500713 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:51.500540    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9155b9e0-f958-4865-a2a2-6efdaa7e9f2f-web-config\") pod \"prometheus-k8s-0\" (UID: \"9155b9e0-f958-4865-a2a2-6efdaa7e9f2f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:25:51.500713 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:51.500563    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9155b9e0-f958-4865-a2a2-6efdaa7e9f2f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"9155b9e0-f958-4865-a2a2-6efdaa7e9f2f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:25:51.500713 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:51.500588    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbbrc\" (UniqueName: \"kubernetes.io/projected/9155b9e0-f958-4865-a2a2-6efdaa7e9f2f-kube-api-access-pbbrc\") pod \"prometheus-k8s-0\" (UID: \"9155b9e0-f958-4865-a2a2-6efdaa7e9f2f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:25:51.501084 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:51.500769    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9155b9e0-f958-4865-a2a2-6efdaa7e9f2f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9155b9e0-f958-4865-a2a2-6efdaa7e9f2f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:25:51.501084 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:51.500813    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9155b9e0-f958-4865-a2a2-6efdaa7e9f2f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"9155b9e0-f958-4865-a2a2-6efdaa7e9f2f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:25:51.501084 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:51.500893    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9155b9e0-f958-4865-a2a2-6efdaa7e9f2f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"9155b9e0-f958-4865-a2a2-6efdaa7e9f2f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:25:51.501084 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:51.500917    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9155b9e0-f958-4865-a2a2-6efdaa7e9f2f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9155b9e0-f958-4865-a2a2-6efdaa7e9f2f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:25:51.501084 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:51.500948    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9155b9e0-f958-4865-a2a2-6efdaa7e9f2f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"9155b9e0-f958-4865-a2a2-6efdaa7e9f2f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:25:51.501084 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:51.500971    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9155b9e0-f958-4865-a2a2-6efdaa7e9f2f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"9155b9e0-f958-4865-a2a2-6efdaa7e9f2f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:25:51.501084 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:51.501009    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9155b9e0-f958-4865-a2a2-6efdaa7e9f2f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9155b9e0-f958-4865-a2a2-6efdaa7e9f2f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:25:51.602476 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:51.602438    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9155b9e0-f958-4865-a2a2-6efdaa7e9f2f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"9155b9e0-f958-4865-a2a2-6efdaa7e9f2f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:25:51.602476 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:51.602481    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9155b9e0-f958-4865-a2a2-6efdaa7e9f2f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"9155b9e0-f958-4865-a2a2-6efdaa7e9f2f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:25:51.602752 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:51.602510    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/9155b9e0-f958-4865-a2a2-6efdaa7e9f2f-config\") pod \"prometheus-k8s-0\" (UID: \"9155b9e0-f958-4865-a2a2-6efdaa7e9f2f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:25:51.602752 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:51.602550    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9155b9e0-f958-4865-a2a2-6efdaa7e9f2f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"9155b9e0-f958-4865-a2a2-6efdaa7e9f2f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:25:51.602752 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:51.602574    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9155b9e0-f958-4865-a2a2-6efdaa7e9f2f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"9155b9e0-f958-4865-a2a2-6efdaa7e9f2f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:25:51.603055 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:51.603022    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9155b9e0-f958-4865-a2a2-6efdaa7e9f2f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"9155b9e0-f958-4865-a2a2-6efdaa7e9f2f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:25:51.603140 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:51.603091    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9155b9e0-f958-4865-a2a2-6efdaa7e9f2f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"9155b9e0-f958-4865-a2a2-6efdaa7e9f2f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:25:51.603140 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:51.603119    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9155b9e0-f958-4865-a2a2-6efdaa7e9f2f-config-out\") pod \"prometheus-k8s-0\" (UID: \"9155b9e0-f958-4865-a2a2-6efdaa7e9f2f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:25:51.603246 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:51.603142    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9155b9e0-f958-4865-a2a2-6efdaa7e9f2f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"9155b9e0-f958-4865-a2a2-6efdaa7e9f2f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:25:51.603246 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:51.603176    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9155b9e0-f958-4865-a2a2-6efdaa7e9f2f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"9155b9e0-f958-4865-a2a2-6efdaa7e9f2f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:25:51.603246 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:51.603201    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9155b9e0-f958-4865-a2a2-6efdaa7e9f2f-web-config\") pod \"prometheus-k8s-0\" (UID: \"9155b9e0-f958-4865-a2a2-6efdaa7e9f2f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:25:51.603246 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:51.603223    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9155b9e0-f958-4865-a2a2-6efdaa7e9f2f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"9155b9e0-f958-4865-a2a2-6efdaa7e9f2f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:25:51.603441 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:51.603253    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbbrc\" (UniqueName: \"kubernetes.io/projected/9155b9e0-f958-4865-a2a2-6efdaa7e9f2f-kube-api-access-pbbrc\") pod \"prometheus-k8s-0\" (UID: \"9155b9e0-f958-4865-a2a2-6efdaa7e9f2f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:25:51.603441 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:51.603333    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3be44d9c-34b9-4f83-b328-de358e619205-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-m2dqb\" (UID: \"3be44d9c-34b9-4f83-b328-de358e619205\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-m2dqb"
Jul 06 20:25:51.603441 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:51.603366    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9155b9e0-f958-4865-a2a2-6efdaa7e9f2f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9155b9e0-f958-4865-a2a2-6efdaa7e9f2f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:25:51.603441 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:51.603402    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9155b9e0-f958-4865-a2a2-6efdaa7e9f2f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"9155b9e0-f958-4865-a2a2-6efdaa7e9f2f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:25:51.603639 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:51.603456    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9155b9e0-f958-4865-a2a2-6efdaa7e9f2f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"9155b9e0-f958-4865-a2a2-6efdaa7e9f2f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:25:51.603639 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:51.603481    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9155b9e0-f958-4865-a2a2-6efdaa7e9f2f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9155b9e0-f958-4865-a2a2-6efdaa7e9f2f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:25:51.603639 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:51.603513    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9155b9e0-f958-4865-a2a2-6efdaa7e9f2f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"9155b9e0-f958-4865-a2a2-6efdaa7e9f2f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:25:51.603639 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:51.603539    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9155b9e0-f958-4865-a2a2-6efdaa7e9f2f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"9155b9e0-f958-4865-a2a2-6efdaa7e9f2f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:25:51.603639 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:51.603574    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9155b9e0-f958-4865-a2a2-6efdaa7e9f2f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9155b9e0-f958-4865-a2a2-6efdaa7e9f2f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:25:51.603639 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:51.603611    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9155b9e0-f958-4865-a2a2-6efdaa7e9f2f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"9155b9e0-f958-4865-a2a2-6efdaa7e9f2f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:25:51.605033 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:51.605002    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9155b9e0-f958-4865-a2a2-6efdaa7e9f2f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9155b9e0-f958-4865-a2a2-6efdaa7e9f2f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:25:51.605405 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:51.605381    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9155b9e0-f958-4865-a2a2-6efdaa7e9f2f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9155b9e0-f958-4865-a2a2-6efdaa7e9f2f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:25:51.605824 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:51.605697    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9155b9e0-f958-4865-a2a2-6efdaa7e9f2f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9155b9e0-f958-4865-a2a2-6efdaa7e9f2f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:25:51.606190 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:51.606169    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9155b9e0-f958-4865-a2a2-6efdaa7e9f2f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"9155b9e0-f958-4865-a2a2-6efdaa7e9f2f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:25:51.606463 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:51.606439    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9155b9e0-f958-4865-a2a2-6efdaa7e9f2f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"9155b9e0-f958-4865-a2a2-6efdaa7e9f2f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:25:51.607779 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:51.606827    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9155b9e0-f958-4865-a2a2-6efdaa7e9f2f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"9155b9e0-f958-4865-a2a2-6efdaa7e9f2f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:25:51.608028 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:51.608003    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9155b9e0-f958-4865-a2a2-6efdaa7e9f2f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"9155b9e0-f958-4865-a2a2-6efdaa7e9f2f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:25:51.608601 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:51.608515    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9155b9e0-f958-4865-a2a2-6efdaa7e9f2f-config-out\") pod \"prometheus-k8s-0\" (UID: \"9155b9e0-f958-4865-a2a2-6efdaa7e9f2f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:25:51.609018 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:51.608930    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/9155b9e0-f958-4865-a2a2-6efdaa7e9f2f-config\") pod \"prometheus-k8s-0\" (UID: \"9155b9e0-f958-4865-a2a2-6efdaa7e9f2f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:25:51.609765 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:51.609671    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9155b9e0-f958-4865-a2a2-6efdaa7e9f2f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"9155b9e0-f958-4865-a2a2-6efdaa7e9f2f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:25:51.610347 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:51.610269    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9155b9e0-f958-4865-a2a2-6efdaa7e9f2f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"9155b9e0-f958-4865-a2a2-6efdaa7e9f2f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:25:51.610926 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:51.610781    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9155b9e0-f958-4865-a2a2-6efdaa7e9f2f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"9155b9e0-f958-4865-a2a2-6efdaa7e9f2f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:25:51.610926 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:51.610876    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9155b9e0-f958-4865-a2a2-6efdaa7e9f2f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"9155b9e0-f958-4865-a2a2-6efdaa7e9f2f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:25:51.611084 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:51.611063    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9155b9e0-f958-4865-a2a2-6efdaa7e9f2f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"9155b9e0-f958-4865-a2a2-6efdaa7e9f2f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:25:51.611147 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:51.611132    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9155b9e0-f958-4865-a2a2-6efdaa7e9f2f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"9155b9e0-f958-4865-a2a2-6efdaa7e9f2f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:25:51.611635 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:51.611580    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3be44d9c-34b9-4f83-b328-de358e619205-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-m2dqb\" (UID: \"3be44d9c-34b9-4f83-b328-de358e619205\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-m2dqb"
Jul 06 20:25:51.611805 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:51.611786    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9155b9e0-f958-4865-a2a2-6efdaa7e9f2f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"9155b9e0-f958-4865-a2a2-6efdaa7e9f2f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:25:51.612229 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:51.612189    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9155b9e0-f958-4865-a2a2-6efdaa7e9f2f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"9155b9e0-f958-4865-a2a2-6efdaa7e9f2f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:25:51.613237 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:51.613200    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9155b9e0-f958-4865-a2a2-6efdaa7e9f2f-web-config\") pod \"prometheus-k8s-0\" (UID: \"9155b9e0-f958-4865-a2a2-6efdaa7e9f2f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:25:51.615784 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:51.615759    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbbrc\" (UniqueName: \"kubernetes.io/projected/9155b9e0-f958-4865-a2a2-6efdaa7e9f2f-kube-api-access-pbbrc\") pod \"prometheus-k8s-0\" (UID: \"9155b9e0-f958-4865-a2a2-6efdaa7e9f2f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:25:51.724296 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:51.724155    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-m2dqb"
Jul 06 20:25:51.766591 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:51.766553    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:25:52.229787 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:52.229668    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 06 20:25:52.231614 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:25:52.231584    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9155b9e0_f958_4865_a2a2_6efdaa7e9f2f.slice/crio-48a6339b21f9ab4e8c2ae9e109dd35955312f6c2395d7dd3702ba7a5017e9b2c WatchSource:0}: Error finding container 48a6339b21f9ab4e8c2ae9e109dd35955312f6c2395d7dd3702ba7a5017e9b2c: Status 404 returned error can't find the container with id 48a6339b21f9ab4e8c2ae9e109dd35955312f6c2395d7dd3702ba7a5017e9b2c
Jul 06 20:25:52.247096 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:52.246984    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-m2dqb"]
Jul 06 20:25:52.416568 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:52.416496    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5b84f65c46-pb5v8" event={"ID":"6827cc99-95c6-4a25-8e25-e73e7e3e2d7c","Type":"ContainerStarted","Data":"276a5a359855be922f783cfd1c487309fb17f5b6d2bb0e2473b41523a8c272da"}
Jul 06 20:25:52.421529 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:52.421453    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f7dadb57-9f0f-4e2d-9a22-0d9e200528e1","Type":"ContainerStarted","Data":"7dfbdf3cc5ca48e8be8ac3e0de69e052966c596d32877facc2c2b145d89057b0"}
Jul 06 20:25:52.421639 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:52.421590    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f7dadb57-9f0f-4e2d-9a22-0d9e200528e1","Type":"ContainerStarted","Data":"dcc62dcfc9cea70070920c70d38b381401b4c9a1f34ed07e043e1d44db32765c"}
Jul 06 20:25:52.421694 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:52.421639    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f7dadb57-9f0f-4e2d-9a22-0d9e200528e1","Type":"ContainerStarted","Data":"8b313feb12d994761055915e8ed981c61fc87492568e9117e2295dd920519eb4"}
Jul 06 20:25:52.422920 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:52.422894    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-m2dqb" event={"ID":"3be44d9c-34b9-4f83-b328-de358e619205","Type":"ContainerStarted","Data":"7b7fe8a7d619093434310f930ba9af88efaed6917ba6672a9fdbde3ed03be6bb"}
Jul 06 20:25:52.425269 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:52.424848    2563 generic.go:356] "Generic (PLEG): container finished" podID="9155b9e0-f958-4865-a2a2-6efdaa7e9f2f" containerID="ea17832b610f65b0a5f658cee6415de6566cd26f82b18903aaccc259587a0c7d" exitCode=0
Jul 06 20:25:52.425269 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:52.424904    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9155b9e0-f958-4865-a2a2-6efdaa7e9f2f","Type":"ContainerDied","Data":"ea17832b610f65b0a5f658cee6415de6566cd26f82b18903aaccc259587a0c7d"}
Jul 06 20:25:52.425269 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:52.424965    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9155b9e0-f958-4865-a2a2-6efdaa7e9f2f","Type":"ContainerStarted","Data":"48a6339b21f9ab4e8c2ae9e109dd35955312f6c2395d7dd3702ba7a5017e9b2c"}
Jul 06 20:25:52.434075 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:52.433881    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-5b84f65c46-pb5v8" podStartSLOduration=1.270159977 podStartE2EDuration="3.433827423s" totalImagesPullingTime="2.163667446s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:25:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:25:52.433025785 +0000 UTC m=+116.149136445" watchObservedRunningTime="2026-07-06 20:25:52.433827423 +0000 UTC m=+116.149938084"
Jul 06 20:25:53.434967 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:53.434921    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f7dadb57-9f0f-4e2d-9a22-0d9e200528e1","Type":"ContainerStarted","Data":"25776ec0fd808c099643abb2b54c1407bea2d3ebb5f044b3b09b8e505d2789c1"}
Jul 06 20:25:53.435422 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:53.435062    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f7dadb57-9f0f-4e2d-9a22-0d9e200528e1","Type":"ContainerStarted","Data":"c67f43b4241a38b17019f3e9a7d4ddcf9e66f44ca3cb8414f463d7dfc87f7efb"}
Jul 06 20:25:53.772188 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:53.772081    2563 patch_prober.go:28] interesting pod/image-registry-7f79498b6c-bkcz7 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 06 20:25:53.772188 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:53.772158    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-7f79498b6c-bkcz7" podUID="b27d84e3-9abb-4322-a9cb-c59f7d85ab10" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 06 20:25:54.614277 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:54.614236    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:25:54.724602 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:54.724576    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:25:55.447409 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:55.447369    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-m2dqb" event={"ID":"3be44d9c-34b9-4f83-b328-de358e619205","Type":"ContainerStarted","Data":"cb46e769f61ff691a5c3b662fdb795d71aaec768f0ed8279de576cfd53cc5334"}
Jul 06 20:25:55.447802 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:55.447641    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-m2dqb"
Jul 06 20:25:55.449530 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:55.449499    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78b5766db6-4r7qh" event={"ID":"74019db4-ab3b-4e74-8cd0-a0fa363732d6","Type":"ContainerStarted","Data":"bafed6c06fbae1fd4e6ac12859132fc4299a1f2f4fc0d6368fdc2f243902e0c9"}
Jul 06 20:25:55.453991 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:55.453962    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f7dadb57-9f0f-4e2d-9a22-0d9e200528e1","Type":"ContainerStarted","Data":"d7f271d1ab6de089d3f09a71cbe7649784050857cd2803cbc50f31e6c2d8bba4"}
Jul 06 20:25:55.454410 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:55.454382    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-m2dqb"
Jul 06 20:25:55.466381 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:55.466334    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-m2dqb" podStartSLOduration=4.084437131 podStartE2EDuration="6.466320411s" totalImagesPullingTime="2.38188328s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:25:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:25:55.464947529 +0000 UTC m=+119.181058201" watchObservedRunningTime="2026-07-06 20:25:55.466320411 +0000 UTC m=+119.182431072"
Jul 06 20:25:55.485764 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:55.485700    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-78b5766db6-4r7qh" podStartSLOduration=2.277220994 podStartE2EDuration="6.485680506s" totalImagesPullingTime="4.208459512s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:25:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:25:55.484256219 +0000 UTC m=+119.200366881" watchObservedRunningTime="2026-07-06 20:25:55.485680506 +0000 UTC m=+119.201791168"
Jul 06 20:25:55.538158 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:55.538088    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.507012845 podStartE2EDuration="9.538066098s" totalImagesPullingTime="6.016493401s" totalInitContainerRuntime="14.559852ms" isStatefulPod=false podCreationTimestamp="2026-07-06 20:25:46 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:25:55.534131415 +0000 UTC m=+119.250242079" watchObservedRunningTime="2026-07-06 20:25:55.538066098 +0000 UTC m=+119.254176758"
Jul 06 20:25:55.751346 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:55.751214    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:25:55.828820 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:55.828775    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:25:55.923360 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:55.923324    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:25:55.992941 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:55.992906    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:25:56.462525 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:56.462490    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9155b9e0-f958-4865-a2a2-6efdaa7e9f2f","Type":"ContainerStarted","Data":"83ee291b1bd70cf6772f865391715d0bd51b3028744df901141d228d5207897a"}
Jul 06 20:25:56.462714 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:56.462589    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9155b9e0-f958-4865-a2a2-6efdaa7e9f2f","Type":"ContainerStarted","Data":"68932e4c162dfa05e9660d65170435619d15d2958f0a09451439be186d3e659b"}
Jul 06 20:25:58.363569 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:58.363536    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:25:58.471550 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:58.471517    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:25:58.794182 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:58.794134    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7f79498b6c-bkcz7" podUID="b27d84e3-9abb-4322-a9cb-c59f7d85ab10" containerName="registry" containerID="cri-o://0307c428fe6aab1962abd7880ad58ebec0734b8c8b7120c443b02613bb54e54d" gracePeriod=30
Jul 06 20:25:59.044696 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:59.044673    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f79498b6c-bkcz7"
Jul 06 20:25:59.191419 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:59.191321    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b27d84e3-9abb-4322-a9cb-c59f7d85ab10-trusted-ca\") pod \"b27d84e3-9abb-4322-a9cb-c59f7d85ab10\" (UID: \"b27d84e3-9abb-4322-a9cb-c59f7d85ab10\") "
Jul 06 20:25:59.191419 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:59.191413    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b27d84e3-9abb-4322-a9cb-c59f7d85ab10-installation-pull-secrets\") pod \"b27d84e3-9abb-4322-a9cb-c59f7d85ab10\" (UID: \"b27d84e3-9abb-4322-a9cb-c59f7d85ab10\") "
Jul 06 20:25:59.191651 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:59.191462    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b27d84e3-9abb-4322-a9cb-c59f7d85ab10-image-registry-private-configuration\") pod \"b27d84e3-9abb-4322-a9cb-c59f7d85ab10\" (UID: \"b27d84e3-9abb-4322-a9cb-c59f7d85ab10\") "
Jul 06 20:25:59.191651 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:59.191485    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b27d84e3-9abb-4322-a9cb-c59f7d85ab10-registry-tls\") pod \"b27d84e3-9abb-4322-a9cb-c59f7d85ab10\" (UID: \"b27d84e3-9abb-4322-a9cb-c59f7d85ab10\") "
Jul 06 20:25:59.191651 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:59.191620    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hzwkh\" (UniqueName: \"kubernetes.io/projected/b27d84e3-9abb-4322-a9cb-c59f7d85ab10-kube-api-access-hzwkh\") pod \"b27d84e3-9abb-4322-a9cb-c59f7d85ab10\" (UID: \"b27d84e3-9abb-4322-a9cb-c59f7d85ab10\") "
Jul 06 20:25:59.191837 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:59.191708    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b27d84e3-9abb-4322-a9cb-c59f7d85ab10-registry-certificates\") pod \"b27d84e3-9abb-4322-a9cb-c59f7d85ab10\" (UID: \"b27d84e3-9abb-4322-a9cb-c59f7d85ab10\") "
Jul 06 20:25:59.191837 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:59.191775    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b27d84e3-9abb-4322-a9cb-c59f7d85ab10-ca-trust-extracted\") pod \"b27d84e3-9abb-4322-a9cb-c59f7d85ab10\" (UID: \"b27d84e3-9abb-4322-a9cb-c59f7d85ab10\") "
Jul 06 20:25:59.191837 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:59.191815    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b27d84e3-9abb-4322-a9cb-c59f7d85ab10-bound-sa-token\") pod \"b27d84e3-9abb-4322-a9cb-c59f7d85ab10\" (UID: \"b27d84e3-9abb-4322-a9cb-c59f7d85ab10\") "
Jul 06 20:25:59.191837 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:59.191824    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b27d84e3-9abb-4322-a9cb-c59f7d85ab10-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "b27d84e3-9abb-4322-a9cb-c59f7d85ab10" (UID: "b27d84e3-9abb-4322-a9cb-c59f7d85ab10"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 20:25:59.192686 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:59.192650    2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b27d84e3-9abb-4322-a9cb-c59f7d85ab10-trusted-ca\") on node \"ip-10-0-141-123.ec2.internal\" DevicePath \"\""
Jul 06 20:25:59.192686 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:59.192655    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b27d84e3-9abb-4322-a9cb-c59f7d85ab10-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "b27d84e3-9abb-4322-a9cb-c59f7d85ab10" (UID: "b27d84e3-9abb-4322-a9cb-c59f7d85ab10"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 20:25:59.194235 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:59.194208    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b27d84e3-9abb-4322-a9cb-c59f7d85ab10-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "b27d84e3-9abb-4322-a9cb-c59f7d85ab10" (UID: "b27d84e3-9abb-4322-a9cb-c59f7d85ab10"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 20:25:59.194353 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:59.194210    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b27d84e3-9abb-4322-a9cb-c59f7d85ab10-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "b27d84e3-9abb-4322-a9cb-c59f7d85ab10" (UID: "b27d84e3-9abb-4322-a9cb-c59f7d85ab10"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 20:25:59.194429 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:59.194342    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b27d84e3-9abb-4322-a9cb-c59f7d85ab10-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "b27d84e3-9abb-4322-a9cb-c59f7d85ab10" (UID: "b27d84e3-9abb-4322-a9cb-c59f7d85ab10"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 20:25:59.194554 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:59.194532    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b27d84e3-9abb-4322-a9cb-c59f7d85ab10-kube-api-access-hzwkh" (OuterVolumeSpecName: "kube-api-access-hzwkh") pod "b27d84e3-9abb-4322-a9cb-c59f7d85ab10" (UID: "b27d84e3-9abb-4322-a9cb-c59f7d85ab10"). InnerVolumeSpecName "kube-api-access-hzwkh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 20:25:59.194781 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:59.194755    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b27d84e3-9abb-4322-a9cb-c59f7d85ab10-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "b27d84e3-9abb-4322-a9cb-c59f7d85ab10" (UID: "b27d84e3-9abb-4322-a9cb-c59f7d85ab10"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 20:25:59.202462 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:59.202439    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b27d84e3-9abb-4322-a9cb-c59f7d85ab10-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "b27d84e3-9abb-4322-a9cb-c59f7d85ab10" (UID: "b27d84e3-9abb-4322-a9cb-c59f7d85ab10"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 20:25:59.238768 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:59.238717    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:25:59.294110 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:59.294078    2563 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b27d84e3-9abb-4322-a9cb-c59f7d85ab10-ca-trust-extracted\") on node \"ip-10-0-141-123.ec2.internal\" DevicePath \"\""
Jul 06 20:25:59.294110 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:59.294102    2563 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b27d84e3-9abb-4322-a9cb-c59f7d85ab10-bound-sa-token\") on node \"ip-10-0-141-123.ec2.internal\" DevicePath \"\""
Jul 06 20:25:59.294110 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:59.294119    2563 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b27d84e3-9abb-4322-a9cb-c59f7d85ab10-installation-pull-secrets\") on node \"ip-10-0-141-123.ec2.internal\" DevicePath \"\""
Jul 06 20:25:59.294332 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:59.294129    2563 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b27d84e3-9abb-4322-a9cb-c59f7d85ab10-image-registry-private-configuration\") on node \"ip-10-0-141-123.ec2.internal\" DevicePath \"\""
Jul 06 20:25:59.294332 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:59.294137    2563 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b27d84e3-9abb-4322-a9cb-c59f7d85ab10-registry-tls\") on node \"ip-10-0-141-123.ec2.internal\" DevicePath \"\""
Jul 06 20:25:59.294332 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:59.294147    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzwkh\" (UniqueName: \"kubernetes.io/projected/b27d84e3-9abb-4322-a9cb-c59f7d85ab10-kube-api-access-hzwkh\") on node \"ip-10-0-141-123.ec2.internal\" DevicePath \"\""
Jul 06 20:25:59.294332 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:59.294156    2563 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b27d84e3-9abb-4322-a9cb-c59f7d85ab10-registry-certificates\") on node \"ip-10-0-141-123.ec2.internal\" DevicePath \"\""
Jul 06 20:25:59.333102 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:59.333065    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:25:59.483259 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:59.483175    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9155b9e0-f958-4865-a2a2-6efdaa7e9f2f","Type":"ContainerStarted","Data":"d4035acf96535ff0faadef756b038c190878f8cc72b2f48b4d2ff1875d417bca"}
Jul 06 20:25:59.483659 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:59.483270    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9155b9e0-f958-4865-a2a2-6efdaa7e9f2f","Type":"ContainerStarted","Data":"ff4662e38b7e5900fd751163f5f7031720db100f1155fcec3c19768e35f9f943"}
Jul 06 20:25:59.483659 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:59.483314    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9155b9e0-f958-4865-a2a2-6efdaa7e9f2f","Type":"ContainerStarted","Data":"78fa493fc6fe6ca1ad69b58cad660058f3adced525935dcd27c04298a0311624"}
Jul 06 20:25:59.483659 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:59.483354    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9155b9e0-f958-4865-a2a2-6efdaa7e9f2f","Type":"ContainerStarted","Data":"9603b18d090f1d1fe2b6df68bd9bff58ec10cf2b479a1f85d3717ee0ab753528"}
Jul 06 20:25:59.484445 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:59.484422    2563 generic.go:356] "Generic (PLEG): container finished" podID="b27d84e3-9abb-4322-a9cb-c59f7d85ab10" containerID="0307c428fe6aab1962abd7880ad58ebec0734b8c8b7120c443b02613bb54e54d" exitCode=0
Jul 06 20:25:59.484534 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:59.484469    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f79498b6c-bkcz7"
Jul 06 20:25:59.484534 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:59.484502    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f79498b6c-bkcz7" event={"ID":"b27d84e3-9abb-4322-a9cb-c59f7d85ab10","Type":"ContainerDied","Data":"0307c428fe6aab1962abd7880ad58ebec0734b8c8b7120c443b02613bb54e54d"}
Jul 06 20:25:59.484607 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:59.484545    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f79498b6c-bkcz7" event={"ID":"b27d84e3-9abb-4322-a9cb-c59f7d85ab10","Type":"ContainerDied","Data":"d192c9b7ad7d2572c1cb907526628589c6015d95c224e22947e0e6ee2bc1a069"}
Jul 06 20:25:59.484607 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:59.484567    2563 scope.go:117] "RemoveContainer" containerID="0307c428fe6aab1962abd7880ad58ebec0734b8c8b7120c443b02613bb54e54d"
Jul 06 20:25:59.493925 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:59.493908    2563 scope.go:117] "RemoveContainer" containerID="0307c428fe6aab1962abd7880ad58ebec0734b8c8b7120c443b02613bb54e54d"
Jul 06 20:25:59.494214 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:25:59.494195    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0307c428fe6aab1962abd7880ad58ebec0734b8c8b7120c443b02613bb54e54d\": container with ID starting with 0307c428fe6aab1962abd7880ad58ebec0734b8c8b7120c443b02613bb54e54d not found: ID does not exist" containerID="0307c428fe6aab1962abd7880ad58ebec0734b8c8b7120c443b02613bb54e54d"
Jul 06 20:25:59.494256 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:59.494222    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0307c428fe6aab1962abd7880ad58ebec0734b8c8b7120c443b02613bb54e54d"} err="failed to get container status \"0307c428fe6aab1962abd7880ad58ebec0734b8c8b7120c443b02613bb54e54d\": rpc error: code = NotFound desc = could not find container \"0307c428fe6aab1962abd7880ad58ebec0734b8c8b7120c443b02613bb54e54d\": container with ID starting with 0307c428fe6aab1962abd7880ad58ebec0734b8c8b7120c443b02613bb54e54d not found: ID does not exist"
Jul 06 20:25:59.554052 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:59.553989    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.709129561 podStartE2EDuration="8.553970211s" totalImagesPullingTime="5.820148707s" totalInitContainerRuntime="24.691943ms" isStatefulPod=false podCreationTimestamp="2026-07-06 20:25:51 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:25:59.551616755 +0000 UTC m=+123.267727428" watchObservedRunningTime="2026-07-06 20:25:59.553970211 +0000 UTC m=+123.270080873"
Jul 06 20:25:59.573327 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:59.573296    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7f79498b6c-bkcz7"]
Jul 06 20:25:59.582145 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:25:59.582123    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7f79498b6c-bkcz7"]
Jul 06 20:26:00.268983 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:26:00.268947    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-78b5766db6-4r7qh"
Jul 06 20:26:00.269154 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:26:00.269016    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-78b5766db6-4r7qh"
Jul 06 20:26:00.273819 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:26:00.273799    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-78b5766db6-4r7qh"
Jul 06 20:26:00.492959 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:26:00.492926    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-78b5766db6-4r7qh"
Jul 06 20:26:00.517971 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:26:00.517941    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:26:00.665497 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:26:00.665462    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:26:00.833354 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:26:00.833319    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b27d84e3-9abb-4322-a9cb-c59f7d85ab10" path="/var/lib/kubelet/pods/b27d84e3-9abb-4322-a9cb-c59f7d85ab10/volumes"
Jul 06 20:26:00.855357 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:26:00.855327    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:26:01.109578 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:26:01.109546    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:26:01.211973 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:26:01.211939    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:26:01.767906 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:26:01.767160    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:26:03.686153 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:26:03.686124    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:26:03.770881 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:26:03.770841    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:26:05.550154 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:26:05.550125    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:26:06.112976 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:26:06.112948    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:26:06.217993 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:26:06.217951    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:26:06.668299 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:26:06.668255    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b20534ab-b8c8-4c81-a0f4-0f98a738896e-metrics-certs\") pod \"network-metrics-daemon-gvd8v\" (UID: \"b20534ab-b8c8-4c81-a0f4-0f98a738896e\") " pod="openshift-multus/network-metrics-daemon-gvd8v"
Jul 06 20:26:06.670764 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:26:06.670722    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b20534ab-b8c8-4c81-a0f4-0f98a738896e-metrics-certs\") pod \"network-metrics-daemon-gvd8v\" (UID: \"b20534ab-b8c8-4c81-a0f4-0f98a738896e\") " pod="openshift-multus/network-metrics-daemon-gvd8v"
Jul 06 20:26:06.820631 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:26:06.820604    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:26:06.856401 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:26:06.856371    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-z7zd8\""
Jul 06 20:26:06.864625 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:26:06.864599    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gvd8v"
Jul 06 20:26:07.015171 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:26:07.015147    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-gvd8v"]
Jul 06 20:26:07.017202 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:26:07.017174    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb20534ab_b8c8_4c81_a0f4_0f98a738896e.slice/crio-7fd05e1b072b49158bdf9bc4719e0e12f1d6fb31b8006d43cb3252b1492f9d99 WatchSource:0}: Error finding container 7fd05e1b072b49158bdf9bc4719e0e12f1d6fb31b8006d43cb3252b1492f9d99: Status 404 returned error can't find the container with id 7fd05e1b072b49158bdf9bc4719e0e12f1d6fb31b8006d43cb3252b1492f9d99
Jul 06 20:26:07.522752 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:26:07.522703    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-gvd8v" event={"ID":"b20534ab-b8c8-4c81-a0f4-0f98a738896e","Type":"ContainerStarted","Data":"7fd05e1b072b49158bdf9bc4719e0e12f1d6fb31b8006d43cb3252b1492f9d99"}
Jul 06 20:26:08.528264 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:26:08.528228    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-gvd8v" event={"ID":"b20534ab-b8c8-4c81-a0f4-0f98a738896e","Type":"ContainerStarted","Data":"db979c3bfa3a2ced0b049506532a10fb91b8ae0c07421440d4653ac245142a4f"}
Jul 06 20:26:08.528634 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:26:08.528288    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-gvd8v" event={"ID":"b20534ab-b8c8-4c81-a0f4-0f98a738896e","Type":"ContainerStarted","Data":"e81ce1b147cb816d8d9526052cf3b2c9a2534683b0b8f401ffec3db2718c8543"}
Jul 06 20:26:08.550268 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:26:08.550223    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-gvd8v" podStartSLOduration=130.553694405 podStartE2EDuration="2m11.550208377s" totalImagesPullingTime="996.513972ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:23:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:26:08.549612516 +0000 UTC m=+132.265723178" watchObservedRunningTime="2026-07-06 20:26:08.550208377 +0000 UTC m=+132.266319038"
Jul 06 20:26:08.689655 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:26:08.689617    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:26:08.781981 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:26:08.781901    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:26:09.746228 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:26:09.746199    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-5b84f65c46-pb5v8"
Jul 06 20:26:09.746587 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:26:09.746254    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-5b84f65c46-pb5v8"
Jul 06 20:26:10.971945 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:26:10.971907    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:26:11.061706 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:26:11.061664    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:26:18.570070 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:26:18.570040    2563 generic.go:356] "Generic (PLEG): container finished" podID="209affe1-4e68-4730-afd0-975faf0bee4c" containerID="b35e6940b121a149690e9983ceddd51f72fbee4a49d7ab7514391201ea29584b" exitCode=0
Jul 06 20:26:18.570471 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:26:18.570111    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-tl6cf" event={"ID":"209affe1-4e68-4730-afd0-975faf0bee4c","Type":"ContainerDied","Data":"b35e6940b121a149690e9983ceddd51f72fbee4a49d7ab7514391201ea29584b"}
Jul 06 20:26:18.570544 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:26:18.570512    2563 scope.go:117] "RemoveContainer" containerID="b35e6940b121a149690e9983ceddd51f72fbee4a49d7ab7514391201ea29584b"
Jul 06 20:26:19.576130 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:26:19.576099    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-tl6cf" event={"ID":"209affe1-4e68-4730-afd0-975faf0bee4c","Type":"ContainerStarted","Data":"7977f0dd18bb7a87fbd7330472f28ec32ae688b4960a89590bb50d1dcc2afd3c"}
Jul 06 20:26:20.554668 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:26:20.554635    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:26:20.617384 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:26:20.617353    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:26:25.102128 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:26:25.102095    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:26:25.166301 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:26:25.166272    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:26:29.752082 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:26:29.752052    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-5b84f65c46-pb5v8"
Jul 06 20:26:29.756492 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:26:29.756462    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-5b84f65c46-pb5v8"
Jul 06 20:26:51.767501 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:26:51.767396    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:26:51.786946 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:26:51.786917    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:26:52.737185 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:26:52.737155    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:26:53.448265 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:26:53.448226    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:27:04.426286 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:04.426242    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:27:04.718292 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:04.718199    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:27:05.500330 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:05.500295    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 06 20:27:05.500855 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:05.500811    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f7dadb57-9f0f-4e2d-9a22-0d9e200528e1" containerName="alertmanager" containerID="cri-o://8b313feb12d994761055915e8ed981c61fc87492568e9117e2295dd920519eb4" gracePeriod=120
Jul 06 20:27:05.500855 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:05.500844    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f7dadb57-9f0f-4e2d-9a22-0d9e200528e1" containerName="prom-label-proxy" containerID="cri-o://d7f271d1ab6de089d3f09a71cbe7649784050857cd2803cbc50f31e6c2d8bba4" gracePeriod=120
Jul 06 20:27:05.500976 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:05.500843    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f7dadb57-9f0f-4e2d-9a22-0d9e200528e1" containerName="kube-rbac-proxy-web" containerID="cri-o://7dfbdf3cc5ca48e8be8ac3e0de69e052966c596d32877facc2c2b145d89057b0" gracePeriod=120
Jul 06 20:27:05.500976 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:05.500876    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f7dadb57-9f0f-4e2d-9a22-0d9e200528e1" containerName="config-reloader" containerID="cri-o://dcc62dcfc9cea70070920c70d38b381401b4c9a1f34ed07e043e1d44db32765c" gracePeriod=120
Jul 06 20:27:05.500976 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:05.500815    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f7dadb57-9f0f-4e2d-9a22-0d9e200528e1" containerName="kube-rbac-proxy-metric" containerID="cri-o://c67f43b4241a38b17019f3e9a7d4ddcf9e66f44ca3cb8414f463d7dfc87f7efb" gracePeriod=120
Jul 06 20:27:05.501113 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:05.501008    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f7dadb57-9f0f-4e2d-9a22-0d9e200528e1" containerName="kube-rbac-proxy" containerID="cri-o://25776ec0fd808c099643abb2b54c1407bea2d3ebb5f044b3b09b8e505d2789c1" gracePeriod=120
Jul 06 20:27:05.779238 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:05.779165    2563 generic.go:356] "Generic (PLEG): container finished" podID="f7dadb57-9f0f-4e2d-9a22-0d9e200528e1" containerID="d7f271d1ab6de089d3f09a71cbe7649784050857cd2803cbc50f31e6c2d8bba4" exitCode=0
Jul 06 20:27:05.779238 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:05.779187    2563 generic.go:356] "Generic (PLEG): container finished" podID="f7dadb57-9f0f-4e2d-9a22-0d9e200528e1" containerID="25776ec0fd808c099643abb2b54c1407bea2d3ebb5f044b3b09b8e505d2789c1" exitCode=0
Jul 06 20:27:05.779238 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:05.779194    2563 generic.go:356] "Generic (PLEG): container finished" podID="f7dadb57-9f0f-4e2d-9a22-0d9e200528e1" containerID="dcc62dcfc9cea70070920c70d38b381401b4c9a1f34ed07e043e1d44db32765c" exitCode=0
Jul 06 20:27:05.779238 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:05.779199    2563 generic.go:356] "Generic (PLEG): container finished" podID="f7dadb57-9f0f-4e2d-9a22-0d9e200528e1" containerID="8b313feb12d994761055915e8ed981c61fc87492568e9117e2295dd920519eb4" exitCode=0
Jul 06 20:27:05.779514 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:05.779238    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f7dadb57-9f0f-4e2d-9a22-0d9e200528e1","Type":"ContainerDied","Data":"d7f271d1ab6de089d3f09a71cbe7649784050857cd2803cbc50f31e6c2d8bba4"}
Jul 06 20:27:05.779514 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:05.779321    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f7dadb57-9f0f-4e2d-9a22-0d9e200528e1","Type":"ContainerDied","Data":"25776ec0fd808c099643abb2b54c1407bea2d3ebb5f044b3b09b8e505d2789c1"}
Jul 06 20:27:05.779514 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:05.779363    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f7dadb57-9f0f-4e2d-9a22-0d9e200528e1","Type":"ContainerDied","Data":"dcc62dcfc9cea70070920c70d38b381401b4c9a1f34ed07e043e1d44db32765c"}
Jul 06 20:27:05.779514 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:05.779416    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f7dadb57-9f0f-4e2d-9a22-0d9e200528e1","Type":"ContainerDied","Data":"8b313feb12d994761055915e8ed981c61fc87492568e9117e2295dd920519eb4"}
Jul 06 20:27:06.737922 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:06.737898    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:27:06.785994 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:06.785963    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f7dadb57-9f0f-4e2d-9a22-0d9e200528e1-cluster-tls-config\") pod \"f7dadb57-9f0f-4e2d-9a22-0d9e200528e1\" (UID: \"f7dadb57-9f0f-4e2d-9a22-0d9e200528e1\") "
Jul 06 20:27:06.786465 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:06.786443    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f7dadb57-9f0f-4e2d-9a22-0d9e200528e1-secret-alertmanager-kube-rbac-proxy-metric\") pod \"f7dadb57-9f0f-4e2d-9a22-0d9e200528e1\" (UID: \"f7dadb57-9f0f-4e2d-9a22-0d9e200528e1\") "
Jul 06 20:27:06.786653 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:06.786624    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hvm2k\" (UniqueName: \"kubernetes.io/projected/f7dadb57-9f0f-4e2d-9a22-0d9e200528e1-kube-api-access-hvm2k\") pod \"f7dadb57-9f0f-4e2d-9a22-0d9e200528e1\" (UID: \"f7dadb57-9f0f-4e2d-9a22-0d9e200528e1\") "
Jul 06 20:27:06.786791 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:06.786697    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f7dadb57-9f0f-4e2d-9a22-0d9e200528e1-secret-alertmanager-kube-rbac-proxy-web\") pod \"f7dadb57-9f0f-4e2d-9a22-0d9e200528e1\" (UID: \"f7dadb57-9f0f-4e2d-9a22-0d9e200528e1\") "
Jul 06 20:27:06.786791 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:06.786749    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f7dadb57-9f0f-4e2d-9a22-0d9e200528e1-web-config\") pod \"f7dadb57-9f0f-4e2d-9a22-0d9e200528e1\" (UID: \"f7dadb57-9f0f-4e2d-9a22-0d9e200528e1\") "
Jul 06 20:27:06.786791 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:06.786779    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f7dadb57-9f0f-4e2d-9a22-0d9e200528e1-tls-assets\") pod \"f7dadb57-9f0f-4e2d-9a22-0d9e200528e1\" (UID: \"f7dadb57-9f0f-4e2d-9a22-0d9e200528e1\") "
Jul 06 20:27:06.786973 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:06.786813    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f7dadb57-9f0f-4e2d-9a22-0d9e200528e1-config-volume\") pod \"f7dadb57-9f0f-4e2d-9a22-0d9e200528e1\" (UID: \"f7dadb57-9f0f-4e2d-9a22-0d9e200528e1\") "
Jul 06 20:27:06.786973 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:06.786852    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f7dadb57-9f0f-4e2d-9a22-0d9e200528e1-config-out\") pod \"f7dadb57-9f0f-4e2d-9a22-0d9e200528e1\" (UID: \"f7dadb57-9f0f-4e2d-9a22-0d9e200528e1\") "
Jul 06 20:27:06.786973 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:06.786880    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f7dadb57-9f0f-4e2d-9a22-0d9e200528e1-alertmanager-main-db\") pod \"f7dadb57-9f0f-4e2d-9a22-0d9e200528e1\" (UID: \"f7dadb57-9f0f-4e2d-9a22-0d9e200528e1\") "
Jul 06 20:27:06.786973 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:06.786907    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f7dadb57-9f0f-4e2d-9a22-0d9e200528e1-metrics-client-ca\") pod \"f7dadb57-9f0f-4e2d-9a22-0d9e200528e1\" (UID: \"f7dadb57-9f0f-4e2d-9a22-0d9e200528e1\") "
Jul 06 20:27:06.786973 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:06.786930    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7dadb57-9f0f-4e2d-9a22-0d9e200528e1-alertmanager-trusted-ca-bundle\") pod \"f7dadb57-9f0f-4e2d-9a22-0d9e200528e1\" (UID: \"f7dadb57-9f0f-4e2d-9a22-0d9e200528e1\") "
Jul 06 20:27:06.786973 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:06.786959    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f7dadb57-9f0f-4e2d-9a22-0d9e200528e1-secret-alertmanager-main-tls\") pod \"f7dadb57-9f0f-4e2d-9a22-0d9e200528e1\" (UID: \"f7dadb57-9f0f-4e2d-9a22-0d9e200528e1\") "
Jul 06 20:27:06.787286 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:06.787012    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f7dadb57-9f0f-4e2d-9a22-0d9e200528e1-secret-alertmanager-kube-rbac-proxy\") pod \"f7dadb57-9f0f-4e2d-9a22-0d9e200528e1\" (UID: \"f7dadb57-9f0f-4e2d-9a22-0d9e200528e1\") "
Jul 06 20:27:06.788310 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:06.787695    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7dadb57-9f0f-4e2d-9a22-0d9e200528e1-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "f7dadb57-9f0f-4e2d-9a22-0d9e200528e1" (UID: "f7dadb57-9f0f-4e2d-9a22-0d9e200528e1"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 20:27:06.788310 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:06.788019    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7dadb57-9f0f-4e2d-9a22-0d9e200528e1-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "f7dadb57-9f0f-4e2d-9a22-0d9e200528e1" (UID: "f7dadb57-9f0f-4e2d-9a22-0d9e200528e1"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 20:27:06.788461 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:06.788327    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7dadb57-9f0f-4e2d-9a22-0d9e200528e1-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "f7dadb57-9f0f-4e2d-9a22-0d9e200528e1" (UID: "f7dadb57-9f0f-4e2d-9a22-0d9e200528e1"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 20:27:06.789634 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:06.789584    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7dadb57-9f0f-4e2d-9a22-0d9e200528e1-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "f7dadb57-9f0f-4e2d-9a22-0d9e200528e1" (UID: "f7dadb57-9f0f-4e2d-9a22-0d9e200528e1"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 20:27:06.790458 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:06.790409    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7dadb57-9f0f-4e2d-9a22-0d9e200528e1-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "f7dadb57-9f0f-4e2d-9a22-0d9e200528e1" (UID: "f7dadb57-9f0f-4e2d-9a22-0d9e200528e1"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 20:27:06.792433 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:06.792399    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7dadb57-9f0f-4e2d-9a22-0d9e200528e1-kube-api-access-hvm2k" (OuterVolumeSpecName: "kube-api-access-hvm2k") pod "f7dadb57-9f0f-4e2d-9a22-0d9e200528e1" (UID: "f7dadb57-9f0f-4e2d-9a22-0d9e200528e1"). InnerVolumeSpecName "kube-api-access-hvm2k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 20:27:06.793043 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:06.792999    2563 generic.go:356] "Generic (PLEG): container finished" podID="f7dadb57-9f0f-4e2d-9a22-0d9e200528e1" containerID="c67f43b4241a38b17019f3e9a7d4ddcf9e66f44ca3cb8414f463d7dfc87f7efb" exitCode=0
Jul 06 20:27:06.793043 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:06.793031    2563 generic.go:356] "Generic (PLEG): container finished" podID="f7dadb57-9f0f-4e2d-9a22-0d9e200528e1" containerID="7dfbdf3cc5ca48e8be8ac3e0de69e052966c596d32877facc2c2b145d89057b0" exitCode=0
Jul 06 20:27:06.793189 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:06.793085    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f7dadb57-9f0f-4e2d-9a22-0d9e200528e1","Type":"ContainerDied","Data":"c67f43b4241a38b17019f3e9a7d4ddcf9e66f44ca3cb8414f463d7dfc87f7efb"}
Jul 06 20:27:06.793189 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:06.793161    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f7dadb57-9f0f-4e2d-9a22-0d9e200528e1","Type":"ContainerDied","Data":"7dfbdf3cc5ca48e8be8ac3e0de69e052966c596d32877facc2c2b145d89057b0"}
Jul 06 20:27:06.793289 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:06.793203    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f7dadb57-9f0f-4e2d-9a22-0d9e200528e1","Type":"ContainerDied","Data":"837c9068e192a0ef36f6dc32a898f081f20d53dd3e9518a76b80e5601be31d1f"}
Jul 06 20:27:06.793289 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:06.793249    2563 scope.go:117] "RemoveContainer" containerID="d7f271d1ab6de089d3f09a71cbe7649784050857cd2803cbc50f31e6c2d8bba4"
Jul 06 20:27:06.793473 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:06.793454    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:27:06.793610 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:06.793587    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7dadb57-9f0f-4e2d-9a22-0d9e200528e1-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "f7dadb57-9f0f-4e2d-9a22-0d9e200528e1" (UID: "f7dadb57-9f0f-4e2d-9a22-0d9e200528e1"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 20:27:06.794040 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:06.794003    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7dadb57-9f0f-4e2d-9a22-0d9e200528e1-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "f7dadb57-9f0f-4e2d-9a22-0d9e200528e1" (UID: "f7dadb57-9f0f-4e2d-9a22-0d9e200528e1"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 20:27:06.794148 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:06.794072    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7dadb57-9f0f-4e2d-9a22-0d9e200528e1-config-volume" (OuterVolumeSpecName: "config-volume") pod "f7dadb57-9f0f-4e2d-9a22-0d9e200528e1" (UID: "f7dadb57-9f0f-4e2d-9a22-0d9e200528e1"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 20:27:06.795242 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:06.794412    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7dadb57-9f0f-4e2d-9a22-0d9e200528e1-config-out" (OuterVolumeSpecName: "config-out") pod "f7dadb57-9f0f-4e2d-9a22-0d9e200528e1" (UID: "f7dadb57-9f0f-4e2d-9a22-0d9e200528e1"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 20:27:06.795242 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:06.794582    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7dadb57-9f0f-4e2d-9a22-0d9e200528e1-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "f7dadb57-9f0f-4e2d-9a22-0d9e200528e1" (UID: "f7dadb57-9f0f-4e2d-9a22-0d9e200528e1"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 20:27:06.798828 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:06.798747    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7dadb57-9f0f-4e2d-9a22-0d9e200528e1-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "f7dadb57-9f0f-4e2d-9a22-0d9e200528e1" (UID: "f7dadb57-9f0f-4e2d-9a22-0d9e200528e1"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 20:27:06.803409 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:06.803390    2563 scope.go:117] "RemoveContainer" containerID="c67f43b4241a38b17019f3e9a7d4ddcf9e66f44ca3cb8414f463d7dfc87f7efb"
Jul 06 20:27:06.808247 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:06.808223    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7dadb57-9f0f-4e2d-9a22-0d9e200528e1-web-config" (OuterVolumeSpecName: "web-config") pod "f7dadb57-9f0f-4e2d-9a22-0d9e200528e1" (UID: "f7dadb57-9f0f-4e2d-9a22-0d9e200528e1"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 20:27:06.811005 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:06.810989    2563 scope.go:117] "RemoveContainer" containerID="25776ec0fd808c099643abb2b54c1407bea2d3ebb5f044b3b09b8e505d2789c1"
Jul 06 20:27:06.818277 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:06.818263    2563 scope.go:117] "RemoveContainer" containerID="7dfbdf3cc5ca48e8be8ac3e0de69e052966c596d32877facc2c2b145d89057b0"
Jul 06 20:27:06.824895 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:06.824882    2563 scope.go:117] "RemoveContainer" containerID="dcc62dcfc9cea70070920c70d38b381401b4c9a1f34ed07e043e1d44db32765c"
Jul 06 20:27:06.832518 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:06.832499    2563 scope.go:117] "RemoveContainer" containerID="8b313feb12d994761055915e8ed981c61fc87492568e9117e2295dd920519eb4"
Jul 06 20:27:06.840959 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:06.840945    2563 scope.go:117] "RemoveContainer" containerID="e20b57a7309d0963ab9a112ac25f4fdf8baec86897f4fd9af4cea690f40f382b"
Jul 06 20:27:06.847755 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:06.847723    2563 scope.go:117] "RemoveContainer" containerID="d7f271d1ab6de089d3f09a71cbe7649784050857cd2803cbc50f31e6c2d8bba4"
Jul 06 20:27:06.848002 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:27:06.847984    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7f271d1ab6de089d3f09a71cbe7649784050857cd2803cbc50f31e6c2d8bba4\": container with ID starting with d7f271d1ab6de089d3f09a71cbe7649784050857cd2803cbc50f31e6c2d8bba4 not found: ID does not exist" containerID="d7f271d1ab6de089d3f09a71cbe7649784050857cd2803cbc50f31e6c2d8bba4"
Jul 06 20:27:06.848052 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:06.848011    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7f271d1ab6de089d3f09a71cbe7649784050857cd2803cbc50f31e6c2d8bba4"} err="failed to get container status \"d7f271d1ab6de089d3f09a71cbe7649784050857cd2803cbc50f31e6c2d8bba4\": rpc error: code = NotFound desc = could not find container \"d7f271d1ab6de089d3f09a71cbe7649784050857cd2803cbc50f31e6c2d8bba4\": container with ID starting with d7f271d1ab6de089d3f09a71cbe7649784050857cd2803cbc50f31e6c2d8bba4 not found: ID does not exist"
Jul 06 20:27:06.848052 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:06.848029    2563 scope.go:117] "RemoveContainer" containerID="c67f43b4241a38b17019f3e9a7d4ddcf9e66f44ca3cb8414f463d7dfc87f7efb"
Jul 06 20:27:06.848270 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:27:06.848252    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c67f43b4241a38b17019f3e9a7d4ddcf9e66f44ca3cb8414f463d7dfc87f7efb\": container with ID starting with c67f43b4241a38b17019f3e9a7d4ddcf9e66f44ca3cb8414f463d7dfc87f7efb not found: ID does not exist" containerID="c67f43b4241a38b17019f3e9a7d4ddcf9e66f44ca3cb8414f463d7dfc87f7efb"
Jul 06 20:27:06.848322 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:06.848275    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c67f43b4241a38b17019f3e9a7d4ddcf9e66f44ca3cb8414f463d7dfc87f7efb"} err="failed to get container status \"c67f43b4241a38b17019f3e9a7d4ddcf9e66f44ca3cb8414f463d7dfc87f7efb\": rpc error: code = NotFound desc = could not find container \"c67f43b4241a38b17019f3e9a7d4ddcf9e66f44ca3cb8414f463d7dfc87f7efb\": container with ID starting with c67f43b4241a38b17019f3e9a7d4ddcf9e66f44ca3cb8414f463d7dfc87f7efb not found: ID does not exist"
Jul 06 20:27:06.848322 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:06.848290    2563 scope.go:117] "RemoveContainer" containerID="25776ec0fd808c099643abb2b54c1407bea2d3ebb5f044b3b09b8e505d2789c1"
Jul 06 20:27:06.848498 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:27:06.848483    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25776ec0fd808c099643abb2b54c1407bea2d3ebb5f044b3b09b8e505d2789c1\": container with ID starting with 25776ec0fd808c099643abb2b54c1407bea2d3ebb5f044b3b09b8e505d2789c1 not found: ID does not exist" containerID="25776ec0fd808c099643abb2b54c1407bea2d3ebb5f044b3b09b8e505d2789c1"
Jul 06 20:27:06.848532 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:06.848503    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25776ec0fd808c099643abb2b54c1407bea2d3ebb5f044b3b09b8e505d2789c1"} err="failed to get container status \"25776ec0fd808c099643abb2b54c1407bea2d3ebb5f044b3b09b8e505d2789c1\": rpc error: code = NotFound desc = could not find container \"25776ec0fd808c099643abb2b54c1407bea2d3ebb5f044b3b09b8e505d2789c1\": container with ID starting with 25776ec0fd808c099643abb2b54c1407bea2d3ebb5f044b3b09b8e505d2789c1 not found: ID does not exist"
Jul 06 20:27:06.848532 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:06.848516    2563 scope.go:117] "RemoveContainer" containerID="7dfbdf3cc5ca48e8be8ac3e0de69e052966c596d32877facc2c2b145d89057b0"
Jul 06 20:27:06.848723 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:27:06.848708    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7dfbdf3cc5ca48e8be8ac3e0de69e052966c596d32877facc2c2b145d89057b0\": container with ID starting with 7dfbdf3cc5ca48e8be8ac3e0de69e052966c596d32877facc2c2b145d89057b0 not found: ID does not exist" containerID="7dfbdf3cc5ca48e8be8ac3e0de69e052966c596d32877facc2c2b145d89057b0"
Jul 06 20:27:06.848785 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:06.848749    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7dfbdf3cc5ca48e8be8ac3e0de69e052966c596d32877facc2c2b145d89057b0"} err="failed to get container status \"7dfbdf3cc5ca48e8be8ac3e0de69e052966c596d32877facc2c2b145d89057b0\": rpc error: code = NotFound desc = could not find container \"7dfbdf3cc5ca48e8be8ac3e0de69e052966c596d32877facc2c2b145d89057b0\": container with ID starting with 7dfbdf3cc5ca48e8be8ac3e0de69e052966c596d32877facc2c2b145d89057b0 not found: ID does not exist"
Jul 06 20:27:06.848785 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:06.848762    2563 scope.go:117] "RemoveContainer" containerID="dcc62dcfc9cea70070920c70d38b381401b4c9a1f34ed07e043e1d44db32765c"
Jul 06 20:27:06.848952 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:27:06.848936    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dcc62dcfc9cea70070920c70d38b381401b4c9a1f34ed07e043e1d44db32765c\": container with ID starting with dcc62dcfc9cea70070920c70d38b381401b4c9a1f34ed07e043e1d44db32765c not found: ID does not exist" containerID="dcc62dcfc9cea70070920c70d38b381401b4c9a1f34ed07e043e1d44db32765c"
Jul 06 20:27:06.848990 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:06.848956    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcc62dcfc9cea70070920c70d38b381401b4c9a1f34ed07e043e1d44db32765c"} err="failed to get container status \"dcc62dcfc9cea70070920c70d38b381401b4c9a1f34ed07e043e1d44db32765c\": rpc error: code = NotFound desc = could not find container \"dcc62dcfc9cea70070920c70d38b381401b4c9a1f34ed07e043e1d44db32765c\": container with ID starting with dcc62dcfc9cea70070920c70d38b381401b4c9a1f34ed07e043e1d44db32765c not found: ID does not exist"
Jul 06 20:27:06.848990 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:06.848971    2563 scope.go:117] "RemoveContainer" containerID="8b313feb12d994761055915e8ed981c61fc87492568e9117e2295dd920519eb4"
Jul 06 20:27:06.849140 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:27:06.849126    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b313feb12d994761055915e8ed981c61fc87492568e9117e2295dd920519eb4\": container with ID starting with 8b313feb12d994761055915e8ed981c61fc87492568e9117e2295dd920519eb4 not found: ID does not exist" containerID="8b313feb12d994761055915e8ed981c61fc87492568e9117e2295dd920519eb4"
Jul 06 20:27:06.849177 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:06.849145    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b313feb12d994761055915e8ed981c61fc87492568e9117e2295dd920519eb4"} err="failed to get container status \"8b313feb12d994761055915e8ed981c61fc87492568e9117e2295dd920519eb4\": rpc error: code = NotFound desc = could not find container \"8b313feb12d994761055915e8ed981c61fc87492568e9117e2295dd920519eb4\": container with ID starting with 8b313feb12d994761055915e8ed981c61fc87492568e9117e2295dd920519eb4 not found: ID does not exist"
Jul 06 20:27:06.849177 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:06.849157    2563 scope.go:117] "RemoveContainer" containerID="e20b57a7309d0963ab9a112ac25f4fdf8baec86897f4fd9af4cea690f40f382b"
Jul 06 20:27:06.849307 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:27:06.849294    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e20b57a7309d0963ab9a112ac25f4fdf8baec86897f4fd9af4cea690f40f382b\": container with ID starting with e20b57a7309d0963ab9a112ac25f4fdf8baec86897f4fd9af4cea690f40f382b not found: ID does not exist" containerID="e20b57a7309d0963ab9a112ac25f4fdf8baec86897f4fd9af4cea690f40f382b"
Jul 06 20:27:06.849347 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:06.849308    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e20b57a7309d0963ab9a112ac25f4fdf8baec86897f4fd9af4cea690f40f382b"} err="failed to get container status \"e20b57a7309d0963ab9a112ac25f4fdf8baec86897f4fd9af4cea690f40f382b\": rpc error: code = NotFound desc = could not find container \"e20b57a7309d0963ab9a112ac25f4fdf8baec86897f4fd9af4cea690f40f382b\": container with ID starting with e20b57a7309d0963ab9a112ac25f4fdf8baec86897f4fd9af4cea690f40f382b not found: ID does not exist"
Jul 06 20:27:06.849347 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:06.849319    2563 scope.go:117] "RemoveContainer" containerID="d7f271d1ab6de089d3f09a71cbe7649784050857cd2803cbc50f31e6c2d8bba4"
Jul 06 20:27:06.849544 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:06.849524    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7f271d1ab6de089d3f09a71cbe7649784050857cd2803cbc50f31e6c2d8bba4"} err="failed to get container status \"d7f271d1ab6de089d3f09a71cbe7649784050857cd2803cbc50f31e6c2d8bba4\": rpc error: code = NotFound desc = could not find container \"d7f271d1ab6de089d3f09a71cbe7649784050857cd2803cbc50f31e6c2d8bba4\": container with ID starting with d7f271d1ab6de089d3f09a71cbe7649784050857cd2803cbc50f31e6c2d8bba4 not found: ID does not exist"
Jul 06 20:27:06.849589 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:06.849546    2563 scope.go:117] "RemoveContainer" containerID="c67f43b4241a38b17019f3e9a7d4ddcf9e66f44ca3cb8414f463d7dfc87f7efb"
Jul 06 20:27:06.849761 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:06.849738    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c67f43b4241a38b17019f3e9a7d4ddcf9e66f44ca3cb8414f463d7dfc87f7efb"} err="failed to get container status \"c67f43b4241a38b17019f3e9a7d4ddcf9e66f44ca3cb8414f463d7dfc87f7efb\": rpc error: code = NotFound desc = could not find container \"c67f43b4241a38b17019f3e9a7d4ddcf9e66f44ca3cb8414f463d7dfc87f7efb\": container with ID starting with c67f43b4241a38b17019f3e9a7d4ddcf9e66f44ca3cb8414f463d7dfc87f7efb not found: ID does not exist"
Jul 06 20:27:06.849761 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:06.849752    2563 scope.go:117] "RemoveContainer" containerID="25776ec0fd808c099643abb2b54c1407bea2d3ebb5f044b3b09b8e505d2789c1"
Jul 06 20:27:06.849991 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:06.849970    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25776ec0fd808c099643abb2b54c1407bea2d3ebb5f044b3b09b8e505d2789c1"} err="failed to get container status \"25776ec0fd808c099643abb2b54c1407bea2d3ebb5f044b3b09b8e505d2789c1\": rpc error: code = NotFound desc = could not find container \"25776ec0fd808c099643abb2b54c1407bea2d3ebb5f044b3b09b8e505d2789c1\": container with ID starting with 25776ec0fd808c099643abb2b54c1407bea2d3ebb5f044b3b09b8e505d2789c1 not found: ID does not exist"
Jul 06 20:27:06.849991 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:06.849990    2563 scope.go:117] "RemoveContainer" containerID="7dfbdf3cc5ca48e8be8ac3e0de69e052966c596d32877facc2c2b145d89057b0"
Jul 06 20:27:06.850208 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:06.850192    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7dfbdf3cc5ca48e8be8ac3e0de69e052966c596d32877facc2c2b145d89057b0"} err="failed to get container status \"7dfbdf3cc5ca48e8be8ac3e0de69e052966c596d32877facc2c2b145d89057b0\": rpc error: code = NotFound desc = could not find container \"7dfbdf3cc5ca48e8be8ac3e0de69e052966c596d32877facc2c2b145d89057b0\": container with ID starting with 7dfbdf3cc5ca48e8be8ac3e0de69e052966c596d32877facc2c2b145d89057b0 not found: ID does not exist"
Jul 06 20:27:06.850260 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:06.850209    2563 scope.go:117] "RemoveContainer" containerID="dcc62dcfc9cea70070920c70d38b381401b4c9a1f34ed07e043e1d44db32765c"
Jul 06 20:27:06.850393 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:06.850379    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcc62dcfc9cea70070920c70d38b381401b4c9a1f34ed07e043e1d44db32765c"} err="failed to get container status \"dcc62dcfc9cea70070920c70d38b381401b4c9a1f34ed07e043e1d44db32765c\": rpc error: code = NotFound desc = could not find container \"dcc62dcfc9cea70070920c70d38b381401b4c9a1f34ed07e043e1d44db32765c\": container with ID starting with dcc62dcfc9cea70070920c70d38b381401b4c9a1f34ed07e043e1d44db32765c not found: ID does not exist"
Jul 06 20:27:06.850436 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:06.850394    2563 scope.go:117] "RemoveContainer" containerID="8b313feb12d994761055915e8ed981c61fc87492568e9117e2295dd920519eb4"
Jul 06 20:27:06.850569 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:06.850554    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b313feb12d994761055915e8ed981c61fc87492568e9117e2295dd920519eb4"} err="failed to get container status \"8b313feb12d994761055915e8ed981c61fc87492568e9117e2295dd920519eb4\": rpc error: code = NotFound desc = could not find container \"8b313feb12d994761055915e8ed981c61fc87492568e9117e2295dd920519eb4\": container with ID starting with 8b313feb12d994761055915e8ed981c61fc87492568e9117e2295dd920519eb4 not found: ID does not exist"
Jul 06 20:27:06.850609 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:06.850569    2563 scope.go:117] "RemoveContainer" containerID="e20b57a7309d0963ab9a112ac25f4fdf8baec86897f4fd9af4cea690f40f382b"
Jul 06 20:27:06.850759 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:06.850739    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e20b57a7309d0963ab9a112ac25f4fdf8baec86897f4fd9af4cea690f40f382b"} err="failed to get container status \"e20b57a7309d0963ab9a112ac25f4fdf8baec86897f4fd9af4cea690f40f382b\": rpc error: code = NotFound desc = could not find container \"e20b57a7309d0963ab9a112ac25f4fdf8baec86897f4fd9af4cea690f40f382b\": container with ID starting with e20b57a7309d0963ab9a112ac25f4fdf8baec86897f4fd9af4cea690f40f382b not found: ID does not exist"
Jul 06 20:27:06.888275 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:06.888258    2563 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f7dadb57-9f0f-4e2d-9a22-0d9e200528e1-config-volume\") on node \"ip-10-0-141-123.ec2.internal\" DevicePath \"\""
Jul 06 20:27:06.888275 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:06.888276    2563 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f7dadb57-9f0f-4e2d-9a22-0d9e200528e1-config-out\") on node \"ip-10-0-141-123.ec2.internal\" DevicePath \"\""
Jul 06 20:27:06.888401 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:06.888286    2563 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f7dadb57-9f0f-4e2d-9a22-0d9e200528e1-alertmanager-main-db\") on node \"ip-10-0-141-123.ec2.internal\" DevicePath \"\""
Jul 06 20:27:06.888401 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:06.888295    2563 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f7dadb57-9f0f-4e2d-9a22-0d9e200528e1-metrics-client-ca\") on node \"ip-10-0-141-123.ec2.internal\" DevicePath \"\""
Jul 06 20:27:06.888401 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:06.888305    2563 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7dadb57-9f0f-4e2d-9a22-0d9e200528e1-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-141-123.ec2.internal\" DevicePath \"\""
Jul 06 20:27:06.888401 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:06.888314    2563 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f7dadb57-9f0f-4e2d-9a22-0d9e200528e1-secret-alertmanager-main-tls\") on node \"ip-10-0-141-123.ec2.internal\" DevicePath \"\""
Jul 06 20:27:06.888401 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:06.888323    2563 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f7dadb57-9f0f-4e2d-9a22-0d9e200528e1-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-141-123.ec2.internal\" DevicePath \"\""
Jul 06 20:27:06.888401 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:06.888331    2563 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f7dadb57-9f0f-4e2d-9a22-0d9e200528e1-cluster-tls-config\") on node \"ip-10-0-141-123.ec2.internal\" DevicePath \"\""
Jul 06 20:27:06.888401 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:06.888341    2563 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f7dadb57-9f0f-4e2d-9a22-0d9e200528e1-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-141-123.ec2.internal\" DevicePath \"\""
Jul 06 20:27:06.888401 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:06.888350    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvm2k\" (UniqueName: \"kubernetes.io/projected/f7dadb57-9f0f-4e2d-9a22-0d9e200528e1-kube-api-access-hvm2k\") on node \"ip-10-0-141-123.ec2.internal\" DevicePath \"\""
Jul 06 20:27:06.888401 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:06.888358    2563 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f7dadb57-9f0f-4e2d-9a22-0d9e200528e1-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-141-123.ec2.internal\" DevicePath \"\""
Jul 06 20:27:06.888401 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:06.888366    2563 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f7dadb57-9f0f-4e2d-9a22-0d9e200528e1-web-config\") on node \"ip-10-0-141-123.ec2.internal\" DevicePath \"\""
Jul 06 20:27:06.888401 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:06.888374    2563 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f7dadb57-9f0f-4e2d-9a22-0d9e200528e1-tls-assets\") on node \"ip-10-0-141-123.ec2.internal\" DevicePath \"\""
Jul 06 20:27:07.118315 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:07.118231    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 06 20:27:07.127449 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:07.127421    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 06 20:27:07.154466 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:07.154440    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 06 20:27:07.154779 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:07.154722    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7dadb57-9f0f-4e2d-9a22-0d9e200528e1" containerName="init-config-reloader"
Jul 06 20:27:07.154779 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:07.154762    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7dadb57-9f0f-4e2d-9a22-0d9e200528e1" containerName="init-config-reloader"
Jul 06 20:27:07.154779 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:07.154778    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7dadb57-9f0f-4e2d-9a22-0d9e200528e1" containerName="kube-rbac-proxy"
Jul 06 20:27:07.154779 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:07.154783    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7dadb57-9f0f-4e2d-9a22-0d9e200528e1" containerName="kube-rbac-proxy"
Jul 06 20:27:07.154946 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:07.154791    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7dadb57-9f0f-4e2d-9a22-0d9e200528e1" containerName="alertmanager"
Jul 06 20:27:07.154946 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:07.154795    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7dadb57-9f0f-4e2d-9a22-0d9e200528e1" containerName="alertmanager"
Jul 06 20:27:07.154946 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:07.154802    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7dadb57-9f0f-4e2d-9a22-0d9e200528e1" containerName="prom-label-proxy"
Jul 06 20:27:07.154946 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:07.154807    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7dadb57-9f0f-4e2d-9a22-0d9e200528e1" containerName="prom-label-proxy"
Jul 06 20:27:07.154946 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:07.154814    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7dadb57-9f0f-4e2d-9a22-0d9e200528e1" containerName="config-reloader"
Jul 06 20:27:07.154946 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:07.154819    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7dadb57-9f0f-4e2d-9a22-0d9e200528e1" containerName="config-reloader"
Jul 06 20:27:07.154946 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:07.154826    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7dadb57-9f0f-4e2d-9a22-0d9e200528e1" containerName="kube-rbac-proxy-metric"
Jul 06 20:27:07.154946 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:07.154853    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7dadb57-9f0f-4e2d-9a22-0d9e200528e1" containerName="kube-rbac-proxy-metric"
Jul 06 20:27:07.154946 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:07.154864    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7dadb57-9f0f-4e2d-9a22-0d9e200528e1" containerName="kube-rbac-proxy-web"
Jul 06 20:27:07.154946 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:07.154869    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7dadb57-9f0f-4e2d-9a22-0d9e200528e1" containerName="kube-rbac-proxy-web"
Jul 06 20:27:07.154946 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:07.154878    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b27d84e3-9abb-4322-a9cb-c59f7d85ab10" containerName="registry"
Jul 06 20:27:07.154946 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:07.154883    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b27d84e3-9abb-4322-a9cb-c59f7d85ab10" containerName="registry"
Jul 06 20:27:07.154946 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:07.154931    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7dadb57-9f0f-4e2d-9a22-0d9e200528e1" containerName="prom-label-proxy"
Jul 06 20:27:07.154946 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:07.154938    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7dadb57-9f0f-4e2d-9a22-0d9e200528e1" containerName="kube-rbac-proxy-web"
Jul 06 20:27:07.154946 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:07.154944    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7dadb57-9f0f-4e2d-9a22-0d9e200528e1" containerName="kube-rbac-proxy-metric"
Jul 06 20:27:07.154946 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:07.154950    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7dadb57-9f0f-4e2d-9a22-0d9e200528e1" containerName="kube-rbac-proxy"
Jul 06 20:27:07.155378 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:07.154955    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7dadb57-9f0f-4e2d-9a22-0d9e200528e1" containerName="config-reloader"
Jul 06 20:27:07.155378 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:07.154960    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7dadb57-9f0f-4e2d-9a22-0d9e200528e1" containerName="alertmanager"
Jul 06 20:27:07.155378 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:07.154965    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="b27d84e3-9abb-4322-a9cb-c59f7d85ab10" containerName="registry"
Jul 06 20:27:07.158581 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:07.158562    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:27:07.160699 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:07.160674    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 06 20:27:07.160940 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:07.160811    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 06 20:27:07.160940 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:07.160817    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 06 20:27:07.161104 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:07.161085    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 06 20:27:07.161199 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:07.161179    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 06 20:27:07.162820 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:07.161579    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 06 20:27:07.163395 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:07.161621    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 06 20:27:07.163395 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:07.161659    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 06 20:27:07.163574 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:07.161709    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-t49cg\""
Jul 06 20:27:07.167598 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:07.167571    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 06 20:27:07.175187 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:07.175167    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 06 20:27:07.190250 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:07.190233    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f571db4e-af94-4e27-9e81-3575f4620e69-web-config\") pod \"alertmanager-main-0\" (UID: \"f571db4e-af94-4e27-9e81-3575f4620e69\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:27:07.190334 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:07.190260    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f571db4e-af94-4e27-9e81-3575f4620e69-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f571db4e-af94-4e27-9e81-3575f4620e69\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:27:07.190334 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:07.190279    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f571db4e-af94-4e27-9e81-3575f4620e69-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f571db4e-af94-4e27-9e81-3575f4620e69\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:27:07.190334 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:07.190304    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f571db4e-af94-4e27-9e81-3575f4620e69-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f571db4e-af94-4e27-9e81-3575f4620e69\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:27:07.190454 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:07.190358    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f571db4e-af94-4e27-9e81-3575f4620e69-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f571db4e-af94-4e27-9e81-3575f4620e69\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:27:07.190454 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:07.190418    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f571db4e-af94-4e27-9e81-3575f4620e69-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f571db4e-af94-4e27-9e81-3575f4620e69\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:27:07.190454 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:07.190439    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f571db4e-af94-4e27-9e81-3575f4620e69-config-volume\") pod \"alertmanager-main-0\" (UID: \"f571db4e-af94-4e27-9e81-3575f4620e69\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:27:07.190655 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:07.190455    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f571db4e-af94-4e27-9e81-3575f4620e69-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f571db4e-af94-4e27-9e81-3575f4620e69\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:27:07.190655 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:07.190472    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f571db4e-af94-4e27-9e81-3575f4620e69-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f571db4e-af94-4e27-9e81-3575f4620e69\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:27:07.190655 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:07.190520    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f571db4e-af94-4e27-9e81-3575f4620e69-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f571db4e-af94-4e27-9e81-3575f4620e69\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:27:07.190655 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:07.190536    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f571db4e-af94-4e27-9e81-3575f4620e69-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f571db4e-af94-4e27-9e81-3575f4620e69\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:27:07.190655 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:07.190588    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxfgb\" (UniqueName: \"kubernetes.io/projected/f571db4e-af94-4e27-9e81-3575f4620e69-kube-api-access-sxfgb\") pod \"alertmanager-main-0\" (UID: \"f571db4e-af94-4e27-9e81-3575f4620e69\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:27:07.190655 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:07.190620    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f571db4e-af94-4e27-9e81-3575f4620e69-config-out\") pod \"alertmanager-main-0\" (UID: \"f571db4e-af94-4e27-9e81-3575f4620e69\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:27:07.291750 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:07.291706    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f571db4e-af94-4e27-9e81-3575f4620e69-web-config\") pod \"alertmanager-main-0\" (UID: \"f571db4e-af94-4e27-9e81-3575f4620e69\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:27:07.291859 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:07.291762    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f571db4e-af94-4e27-9e81-3575f4620e69-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f571db4e-af94-4e27-9e81-3575f4620e69\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:27:07.291859 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:07.291787    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f571db4e-af94-4e27-9e81-3575f4620e69-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f571db4e-af94-4e27-9e81-3575f4620e69\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:27:07.291859 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:07.291820    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f571db4e-af94-4e27-9e81-3575f4620e69-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f571db4e-af94-4e27-9e81-3575f4620e69\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:27:07.291967 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:07.291868    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f571db4e-af94-4e27-9e81-3575f4620e69-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f571db4e-af94-4e27-9e81-3575f4620e69\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:27:07.291967 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:07.291904    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f571db4e-af94-4e27-9e81-3575f4620e69-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f571db4e-af94-4e27-9e81-3575f4620e69\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:27:07.291967 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:07.291926    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f571db4e-af94-4e27-9e81-3575f4620e69-config-volume\") pod \"alertmanager-main-0\" (UID: \"f571db4e-af94-4e27-9e81-3575f4620e69\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:27:07.291967 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:07.291943    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f571db4e-af94-4e27-9e81-3575f4620e69-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f571db4e-af94-4e27-9e81-3575f4620e69\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:27:07.292152 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:07.291967    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f571db4e-af94-4e27-9e81-3575f4620e69-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f571db4e-af94-4e27-9e81-3575f4620e69\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:27:07.292152 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:07.292002    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f571db4e-af94-4e27-9e81-3575f4620e69-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f571db4e-af94-4e27-9e81-3575f4620e69\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:27:07.292152 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:07.292025    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f571db4e-af94-4e27-9e81-3575f4620e69-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f571db4e-af94-4e27-9e81-3575f4620e69\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:27:07.292152 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:07.292064    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxfgb\" (UniqueName: \"kubernetes.io/projected/f571db4e-af94-4e27-9e81-3575f4620e69-kube-api-access-sxfgb\") pod \"alertmanager-main-0\" (UID: \"f571db4e-af94-4e27-9e81-3575f4620e69\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:27:07.292152 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:07.292089    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f571db4e-af94-4e27-9e81-3575f4620e69-config-out\") pod \"alertmanager-main-0\" (UID: \"f571db4e-af94-4e27-9e81-3575f4620e69\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:27:07.292609 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:07.292580    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f571db4e-af94-4e27-9e81-3575f4620e69-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f571db4e-af94-4e27-9e81-3575f4620e69\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:27:07.294429 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:07.294401    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f571db4e-af94-4e27-9e81-3575f4620e69-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f571db4e-af94-4e27-9e81-3575f4620e69\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:27:07.294888 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:07.294861    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f571db4e-af94-4e27-9e81-3575f4620e69-config-out\") pod \"alertmanager-main-0\" (UID: \"f571db4e-af94-4e27-9e81-3575f4620e69\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:27:07.294984 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:07.294968    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f571db4e-af94-4e27-9e81-3575f4620e69-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f571db4e-af94-4e27-9e81-3575f4620e69\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:27:07.295338 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:07.295068    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f571db4e-af94-4e27-9e81-3575f4620e69-web-config\") pod \"alertmanager-main-0\" (UID: \"f571db4e-af94-4e27-9e81-3575f4620e69\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:27:07.295338 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:07.295157    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f571db4e-af94-4e27-9e81-3575f4620e69-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f571db4e-af94-4e27-9e81-3575f4620e69\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:27:07.295338 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:07.295300    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f571db4e-af94-4e27-9e81-3575f4620e69-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f571db4e-af94-4e27-9e81-3575f4620e69\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:27:07.295338 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:07.295307    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f571db4e-af94-4e27-9e81-3575f4620e69-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f571db4e-af94-4e27-9e81-3575f4620e69\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:27:07.295489 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:07.295390    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f571db4e-af94-4e27-9e81-3575f4620e69-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f571db4e-af94-4e27-9e81-3575f4620e69\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:27:07.295489 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:07.295425    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f571db4e-af94-4e27-9e81-3575f4620e69-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f571db4e-af94-4e27-9e81-3575f4620e69\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:27:07.295489 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:07.295439    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f571db4e-af94-4e27-9e81-3575f4620e69-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f571db4e-af94-4e27-9e81-3575f4620e69\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:27:07.296413 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:07.296397    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f571db4e-af94-4e27-9e81-3575f4620e69-config-volume\") pod \"alertmanager-main-0\" (UID: \"f571db4e-af94-4e27-9e81-3575f4620e69\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:27:07.302095 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:07.302079    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxfgb\" (UniqueName: \"kubernetes.io/projected/f571db4e-af94-4e27-9e81-3575f4620e69-kube-api-access-sxfgb\") pod \"alertmanager-main-0\" (UID: \"f571db4e-af94-4e27-9e81-3575f4620e69\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:27:07.470432 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:07.470346    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:27:07.603418 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:07.603256    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 06 20:27:07.606039 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:27:07.606015    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf571db4e_af94_4e27_9e81_3575f4620e69.slice/crio-69a45faa496d293dae7a607bc96de1f4481bddb678c5eb1ec7bc060fc760d20a WatchSource:0}: Error finding container 69a45faa496d293dae7a607bc96de1f4481bddb678c5eb1ec7bc060fc760d20a: Status 404 returned error can't find the container with id 69a45faa496d293dae7a607bc96de1f4481bddb678c5eb1ec7bc060fc760d20a
Jul 06 20:27:07.798953 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:07.798852    2563 generic.go:356] "Generic (PLEG): container finished" podID="f571db4e-af94-4e27-9e81-3575f4620e69" containerID="d044ddfa9835d8705f75e39dcf5be475408af304749ee84c5c9da01692bba694" exitCode=0
Jul 06 20:27:07.798953 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:07.798935    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f571db4e-af94-4e27-9e81-3575f4620e69","Type":"ContainerDied","Data":"d044ddfa9835d8705f75e39dcf5be475408af304749ee84c5c9da01692bba694"}
Jul 06 20:27:07.799438 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:07.799009    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f571db4e-af94-4e27-9e81-3575f4620e69","Type":"ContainerStarted","Data":"69a45faa496d293dae7a607bc96de1f4481bddb678c5eb1ec7bc060fc760d20a"}
Jul 06 20:27:08.807316 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:08.807280    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f571db4e-af94-4e27-9e81-3575f4620e69","Type":"ContainerStarted","Data":"d47991ee0d13c556e48662b29c8f7ac1fdb767f761c291a20a93c0a00c936851"}
Jul 06 20:27:08.807774 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:08.807631    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f571db4e-af94-4e27-9e81-3575f4620e69","Type":"ContainerStarted","Data":"ec2e6cdb6a1d61598e68e6cd3dab2bebd6b7eedac1cfc2ef4a1acf295a44f42d"}
Jul 06 20:27:08.811751 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:08.807901    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f571db4e-af94-4e27-9e81-3575f4620e69","Type":"ContainerStarted","Data":"1ef481b9fcd53ddc69cc372d0528337dc0903c637d8241740a30e4b55c2063ba"}
Jul 06 20:27:08.811751 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:08.807968    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f571db4e-af94-4e27-9e81-3575f4620e69","Type":"ContainerStarted","Data":"6f4f6820c8efbca7cefc3659fbd407a438b0214b8c7d1417cf1b0280276c580c"}
Jul 06 20:27:08.811751 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:08.808024    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f571db4e-af94-4e27-9e81-3575f4620e69","Type":"ContainerStarted","Data":"7f99881acacb3333de83e3e1d499761fd6fed71ffb9c737aea1f41dafc7d58e3"}
Jul 06 20:27:08.811751 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:08.808063    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f571db4e-af94-4e27-9e81-3575f4620e69","Type":"ContainerStarted","Data":"63c2b2a32db09631772ca8790a1aec49fc80fb20f65b050b001a99950c5757c7"}
Jul 06 20:27:08.832356 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:08.832295    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f7dadb57-9f0f-4e2d-9a22-0d9e200528e1" path="/var/lib/kubelet/pods/f7dadb57-9f0f-4e2d-9a22-0d9e200528e1/volumes"
Jul 06 20:27:08.838976 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:08.838937    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=1.824072536 podStartE2EDuration="1.838923851s" totalImagesPullingTime="0s" totalInitContainerRuntime="14.851315ms" isStatefulPod=false podCreationTimestamp="2026-07-06 20:27:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:27:08.83762267 +0000 UTC m=+192.553733371" watchObservedRunningTime="2026-07-06 20:27:08.838923851 +0000 UTC m=+192.555034512"
Jul 06 20:27:09.803973 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:09.803944    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 06 20:27:09.804284 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:09.804233    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="9155b9e0-f958-4865-a2a2-6efdaa7e9f2f" containerName="prometheus" containerID="cri-o://68932e4c162dfa05e9660d65170435619d15d2958f0a09451439be186d3e659b" gracePeriod=600
Jul 06 20:27:09.804284 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:09.804258    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="9155b9e0-f958-4865-a2a2-6efdaa7e9f2f" containerName="thanos-sidecar" containerID="cri-o://d4035acf96535ff0faadef756b038c190878f8cc72b2f48b4d2ff1875d417bca" gracePeriod=600
Jul 06 20:27:09.804457 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:09.804284    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="9155b9e0-f958-4865-a2a2-6efdaa7e9f2f" containerName="kube-rbac-proxy-thanos" containerID="cri-o://ff4662e38b7e5900fd751163f5f7031720db100f1155fcec3c19768e35f9f943" gracePeriod=600
Jul 06 20:27:09.804457 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:09.804299    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="9155b9e0-f958-4865-a2a2-6efdaa7e9f2f" containerName="kube-rbac-proxy" containerID="cri-o://78fa493fc6fe6ca1ad69b58cad660058f3adced525935dcd27c04298a0311624" gracePeriod=600
Jul 06 20:27:09.804457 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:09.804257    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="9155b9e0-f958-4865-a2a2-6efdaa7e9f2f" containerName="config-reloader" containerID="cri-o://83ee291b1bd70cf6772f865391715d0bd51b3028744df901141d228d5207897a" gracePeriod=600
Jul 06 20:27:09.804457 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:09.804275    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="9155b9e0-f958-4865-a2a2-6efdaa7e9f2f" containerName="kube-rbac-proxy-web" containerID="cri-o://9603b18d090f1d1fe2b6df68bd9bff58ec10cf2b479a1f85d3717ee0ab753528" gracePeriod=600
Jul 06 20:27:10.064635 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.064567    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:27:10.122402 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.122366    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9155b9e0-f958-4865-a2a2-6efdaa7e9f2f-prometheus-k8s-db\") pod \"9155b9e0-f958-4865-a2a2-6efdaa7e9f2f\" (UID: \"9155b9e0-f958-4865-a2a2-6efdaa7e9f2f\") "
Jul 06 20:27:10.122563 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.122421    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9155b9e0-f958-4865-a2a2-6efdaa7e9f2f-prometheus-k8s-rulefiles-2\") pod \"9155b9e0-f958-4865-a2a2-6efdaa7e9f2f\" (UID: \"9155b9e0-f958-4865-a2a2-6efdaa7e9f2f\") "
Jul 06 20:27:10.122563 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.122466    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9155b9e0-f958-4865-a2a2-6efdaa7e9f2f-secret-metrics-client-certs\") pod \"9155b9e0-f958-4865-a2a2-6efdaa7e9f2f\" (UID: \"9155b9e0-f958-4865-a2a2-6efdaa7e9f2f\") "
Jul 06 20:27:10.122563 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.122496    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9155b9e0-f958-4865-a2a2-6efdaa7e9f2f-secret-prometheus-k8s-tls\") pod \"9155b9e0-f958-4865-a2a2-6efdaa7e9f2f\" (UID: \"9155b9e0-f958-4865-a2a2-6efdaa7e9f2f\") "
Jul 06 20:27:10.122563 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.122524    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9155b9e0-f958-4865-a2a2-6efdaa7e9f2f-configmap-metrics-client-ca\") pod \"9155b9e0-f958-4865-a2a2-6efdaa7e9f2f\" (UID: \"9155b9e0-f958-4865-a2a2-6efdaa7e9f2f\") "
Jul 06 20:27:10.122563 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.122552    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9155b9e0-f958-4865-a2a2-6efdaa7e9f2f-configmap-kubelet-serving-ca-bundle\") pod \"9155b9e0-f958-4865-a2a2-6efdaa7e9f2f\" (UID: \"9155b9e0-f958-4865-a2a2-6efdaa7e9f2f\") "
Jul 06 20:27:10.122843 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.122605    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9155b9e0-f958-4865-a2a2-6efdaa7e9f2f-tls-assets\") pod \"9155b9e0-f958-4865-a2a2-6efdaa7e9f2f\" (UID: \"9155b9e0-f958-4865-a2a2-6efdaa7e9f2f\") "
Jul 06 20:27:10.122843 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.122631    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9155b9e0-f958-4865-a2a2-6efdaa7e9f2f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"9155b9e0-f958-4865-a2a2-6efdaa7e9f2f\" (UID: \"9155b9e0-f958-4865-a2a2-6efdaa7e9f2f\") "
Jul 06 20:27:10.122843 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.122661    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9155b9e0-f958-4865-a2a2-6efdaa7e9f2f-configmap-serving-certs-ca-bundle\") pod \"9155b9e0-f958-4865-a2a2-6efdaa7e9f2f\" (UID: \"9155b9e0-f958-4865-a2a2-6efdaa7e9f2f\") "
Jul 06 20:27:10.122843 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.122689    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9155b9e0-f958-4865-a2a2-6efdaa7e9f2f-secret-kube-rbac-proxy\") pod \"9155b9e0-f958-4865-a2a2-6efdaa7e9f2f\" (UID: \"9155b9e0-f958-4865-a2a2-6efdaa7e9f2f\") "
Jul 06 20:27:10.122843 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.122720    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9155b9e0-f958-4865-a2a2-6efdaa7e9f2f-secret-grpc-tls\") pod \"9155b9e0-f958-4865-a2a2-6efdaa7e9f2f\" (UID: \"9155b9e0-f958-4865-a2a2-6efdaa7e9f2f\") "
Jul 06 20:27:10.122843 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.122760    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9155b9e0-f958-4865-a2a2-6efdaa7e9f2f-thanos-prometheus-http-client-file\") pod \"9155b9e0-f958-4865-a2a2-6efdaa7e9f2f\" (UID: \"9155b9e0-f958-4865-a2a2-6efdaa7e9f2f\") "
Jul 06 20:27:10.122843 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.122796    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/9155b9e0-f958-4865-a2a2-6efdaa7e9f2f-config\") pod \"9155b9e0-f958-4865-a2a2-6efdaa7e9f2f\" (UID: \"9155b9e0-f958-4865-a2a2-6efdaa7e9f2f\") "
Jul 06 20:27:10.122843 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.122844    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pbbrc\" (UniqueName: \"kubernetes.io/projected/9155b9e0-f958-4865-a2a2-6efdaa7e9f2f-kube-api-access-pbbrc\") pod \"9155b9e0-f958-4865-a2a2-6efdaa7e9f2f\" (UID: \"9155b9e0-f958-4865-a2a2-6efdaa7e9f2f\") "
Jul 06 20:27:10.123222 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.122852    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9155b9e0-f958-4865-a2a2-6efdaa7e9f2f-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "9155b9e0-f958-4865-a2a2-6efdaa7e9f2f" (UID: "9155b9e0-f958-4865-a2a2-6efdaa7e9f2f"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 20:27:10.123222 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.122875    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9155b9e0-f958-4865-a2a2-6efdaa7e9f2f-prometheus-k8s-rulefiles-1\") pod \"9155b9e0-f958-4865-a2a2-6efdaa7e9f2f\" (UID: \"9155b9e0-f958-4865-a2a2-6efdaa7e9f2f\") "
Jul 06 20:27:10.123222 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.122895    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9155b9e0-f958-4865-a2a2-6efdaa7e9f2f-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "9155b9e0-f958-4865-a2a2-6efdaa7e9f2f" (UID: "9155b9e0-f958-4865-a2a2-6efdaa7e9f2f"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 20:27:10.123222 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.122909    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9155b9e0-f958-4865-a2a2-6efdaa7e9f2f-prometheus-trusted-ca-bundle\") pod \"9155b9e0-f958-4865-a2a2-6efdaa7e9f2f\" (UID: \"9155b9e0-f958-4865-a2a2-6efdaa7e9f2f\") "
Jul 06 20:27:10.123222 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.122941    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9155b9e0-f958-4865-a2a2-6efdaa7e9f2f-config-out\") pod \"9155b9e0-f958-4865-a2a2-6efdaa7e9f2f\" (UID: \"9155b9e0-f958-4865-a2a2-6efdaa7e9f2f\") "
Jul 06 20:27:10.123222 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.122966    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9155b9e0-f958-4865-a2a2-6efdaa7e9f2f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"9155b9e0-f958-4865-a2a2-6efdaa7e9f2f\" (UID: \"9155b9e0-f958-4865-a2a2-6efdaa7e9f2f\") "
Jul 06 20:27:10.123222 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.122998    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9155b9e0-f958-4865-a2a2-6efdaa7e9f2f-prometheus-k8s-rulefiles-0\") pod \"9155b9e0-f958-4865-a2a2-6efdaa7e9f2f\" (UID: \"9155b9e0-f958-4865-a2a2-6efdaa7e9f2f\") "
Jul 06 20:27:10.123222 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.123026    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9155b9e0-f958-4865-a2a2-6efdaa7e9f2f-web-config\") pod \"9155b9e0-f958-4865-a2a2-6efdaa7e9f2f\" (UID: \"9155b9e0-f958-4865-a2a2-6efdaa7e9f2f\") "
Jul 06 20:27:10.123612 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.123276    2563 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9155b9e0-f958-4865-a2a2-6efdaa7e9f2f-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-141-123.ec2.internal\" DevicePath \"\""
Jul 06 20:27:10.123612 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.123292    2563 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9155b9e0-f958-4865-a2a2-6efdaa7e9f2f-configmap-metrics-client-ca\") on node \"ip-10-0-141-123.ec2.internal\" DevicePath \"\""
Jul 06 20:27:10.123612 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.123244    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9155b9e0-f958-4865-a2a2-6efdaa7e9f2f-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "9155b9e0-f958-4865-a2a2-6efdaa7e9f2f" (UID: "9155b9e0-f958-4865-a2a2-6efdaa7e9f2f"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 20:27:10.123612 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.123348    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9155b9e0-f958-4865-a2a2-6efdaa7e9f2f-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "9155b9e0-f958-4865-a2a2-6efdaa7e9f2f" (UID: "9155b9e0-f958-4865-a2a2-6efdaa7e9f2f"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 20:27:10.123847 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.123787    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9155b9e0-f958-4865-a2a2-6efdaa7e9f2f-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "9155b9e0-f958-4865-a2a2-6efdaa7e9f2f" (UID: "9155b9e0-f958-4865-a2a2-6efdaa7e9f2f"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 20:27:10.124227 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.124198    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9155b9e0-f958-4865-a2a2-6efdaa7e9f2f-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "9155b9e0-f958-4865-a2a2-6efdaa7e9f2f" (UID: "9155b9e0-f958-4865-a2a2-6efdaa7e9f2f"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 20:27:10.124437 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.124412    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9155b9e0-f958-4865-a2a2-6efdaa7e9f2f-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "9155b9e0-f958-4865-a2a2-6efdaa7e9f2f" (UID: "9155b9e0-f958-4865-a2a2-6efdaa7e9f2f"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 20:27:10.126320 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.126275    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9155b9e0-f958-4865-a2a2-6efdaa7e9f2f-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "9155b9e0-f958-4865-a2a2-6efdaa7e9f2f" (UID: "9155b9e0-f958-4865-a2a2-6efdaa7e9f2f"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 20:27:10.126320 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.126305    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9155b9e0-f958-4865-a2a2-6efdaa7e9f2f-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "9155b9e0-f958-4865-a2a2-6efdaa7e9f2f" (UID: "9155b9e0-f958-4865-a2a2-6efdaa7e9f2f"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 20:27:10.127930 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.127879    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9155b9e0-f958-4865-a2a2-6efdaa7e9f2f-config-out" (OuterVolumeSpecName: "config-out") pod "9155b9e0-f958-4865-a2a2-6efdaa7e9f2f" (UID: "9155b9e0-f958-4865-a2a2-6efdaa7e9f2f"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 20:27:10.127930 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.127893    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9155b9e0-f958-4865-a2a2-6efdaa7e9f2f-config" (OuterVolumeSpecName: "config") pod "9155b9e0-f958-4865-a2a2-6efdaa7e9f2f" (UID: "9155b9e0-f958-4865-a2a2-6efdaa7e9f2f"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 20:27:10.128082 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.127954    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9155b9e0-f958-4865-a2a2-6efdaa7e9f2f-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "9155b9e0-f958-4865-a2a2-6efdaa7e9f2f" (UID: "9155b9e0-f958-4865-a2a2-6efdaa7e9f2f"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 20:27:10.128082 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.128067    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9155b9e0-f958-4865-a2a2-6efdaa7e9f2f-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "9155b9e0-f958-4865-a2a2-6efdaa7e9f2f" (UID: "9155b9e0-f958-4865-a2a2-6efdaa7e9f2f"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 20:27:10.128180 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.128085    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9155b9e0-f958-4865-a2a2-6efdaa7e9f2f-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "9155b9e0-f958-4865-a2a2-6efdaa7e9f2f" (UID: "9155b9e0-f958-4865-a2a2-6efdaa7e9f2f"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 20:27:10.128180 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.128150    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9155b9e0-f958-4865-a2a2-6efdaa7e9f2f-kube-api-access-pbbrc" (OuterVolumeSpecName: "kube-api-access-pbbrc") pod "9155b9e0-f958-4865-a2a2-6efdaa7e9f2f" (UID: "9155b9e0-f958-4865-a2a2-6efdaa7e9f2f"). InnerVolumeSpecName "kube-api-access-pbbrc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 20:27:10.128260 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.128179    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9155b9e0-f958-4865-a2a2-6efdaa7e9f2f-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "9155b9e0-f958-4865-a2a2-6efdaa7e9f2f" (UID: "9155b9e0-f958-4865-a2a2-6efdaa7e9f2f"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 20:27:10.128260 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.128188    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9155b9e0-f958-4865-a2a2-6efdaa7e9f2f-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "9155b9e0-f958-4865-a2a2-6efdaa7e9f2f" (UID: "9155b9e0-f958-4865-a2a2-6efdaa7e9f2f"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 20:27:10.128384 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.128363    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9155b9e0-f958-4865-a2a2-6efdaa7e9f2f-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "9155b9e0-f958-4865-a2a2-6efdaa7e9f2f" (UID: "9155b9e0-f958-4865-a2a2-6efdaa7e9f2f"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 20:27:10.129399 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.129371    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9155b9e0-f958-4865-a2a2-6efdaa7e9f2f-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "9155b9e0-f958-4865-a2a2-6efdaa7e9f2f" (UID: "9155b9e0-f958-4865-a2a2-6efdaa7e9f2f"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 20:27:10.138925 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.138898    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9155b9e0-f958-4865-a2a2-6efdaa7e9f2f-web-config" (OuterVolumeSpecName: "web-config") pod "9155b9e0-f958-4865-a2a2-6efdaa7e9f2f" (UID: "9155b9e0-f958-4865-a2a2-6efdaa7e9f2f"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 20:27:10.224046 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.224020    2563 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9155b9e0-f958-4865-a2a2-6efdaa7e9f2f-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-141-123.ec2.internal\" DevicePath \"\""
Jul 06 20:27:10.224046 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.224044    2563 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9155b9e0-f958-4865-a2a2-6efdaa7e9f2f-prometheus-trusted-ca-bundle\") on node \"ip-10-0-141-123.ec2.internal\" DevicePath \"\""
Jul 06 20:27:10.224192 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.224054    2563 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9155b9e0-f958-4865-a2a2-6efdaa7e9f2f-config-out\") on node \"ip-10-0-141-123.ec2.internal\" DevicePath \"\""
Jul 06 20:27:10.224192 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.224063    2563 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9155b9e0-f958-4865-a2a2-6efdaa7e9f2f-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-141-123.ec2.internal\" DevicePath \"\""
Jul 06 20:27:10.224192 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.224073    2563 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9155b9e0-f958-4865-a2a2-6efdaa7e9f2f-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-141-123.ec2.internal\" DevicePath \"\""
Jul 06 20:27:10.224192 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.224081    2563 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9155b9e0-f958-4865-a2a2-6efdaa7e9f2f-web-config\") on node \"ip-10-0-141-123.ec2.internal\" DevicePath \"\""
Jul 06 20:27:10.224192 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.224089    2563 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9155b9e0-f958-4865-a2a2-6efdaa7e9f2f-prometheus-k8s-db\") on node \"ip-10-0-141-123.ec2.internal\" DevicePath \"\""
Jul 06 20:27:10.224192 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.224097    2563 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9155b9e0-f958-4865-a2a2-6efdaa7e9f2f-secret-metrics-client-certs\") on node \"ip-10-0-141-123.ec2.internal\" DevicePath \"\""
Jul 06 20:27:10.224192 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.224105    2563 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9155b9e0-f958-4865-a2a2-6efdaa7e9f2f-secret-prometheus-k8s-tls\") on node \"ip-10-0-141-123.ec2.internal\" DevicePath \"\""
Jul 06 20:27:10.224192 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.224114    2563 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9155b9e0-f958-4865-a2a2-6efdaa7e9f2f-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-141-123.ec2.internal\" DevicePath \"\""
Jul 06 20:27:10.224192 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.224122    2563 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9155b9e0-f958-4865-a2a2-6efdaa7e9f2f-tls-assets\") on node \"ip-10-0-141-123.ec2.internal\" DevicePath \"\""
Jul 06 20:27:10.224192 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.224130    2563 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9155b9e0-f958-4865-a2a2-6efdaa7e9f2f-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-141-123.ec2.internal\" DevicePath \"\""
Jul 06 20:27:10.224192 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.224138    2563 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9155b9e0-f958-4865-a2a2-6efdaa7e9f2f-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-141-123.ec2.internal\" DevicePath \"\""
Jul 06 20:27:10.224192 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.224147    2563 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9155b9e0-f958-4865-a2a2-6efdaa7e9f2f-secret-kube-rbac-proxy\") on node \"ip-10-0-141-123.ec2.internal\" DevicePath \"\""
Jul 06 20:27:10.224192 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.224154    2563 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9155b9e0-f958-4865-a2a2-6efdaa7e9f2f-secret-grpc-tls\") on node \"ip-10-0-141-123.ec2.internal\" DevicePath \"\""
Jul 06 20:27:10.224192 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.224162    2563 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9155b9e0-f958-4865-a2a2-6efdaa7e9f2f-thanos-prometheus-http-client-file\") on node \"ip-10-0-141-123.ec2.internal\" DevicePath \"\""
Jul 06 20:27:10.224192 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.224170    2563 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/9155b9e0-f958-4865-a2a2-6efdaa7e9f2f-config\") on node \"ip-10-0-141-123.ec2.internal\" DevicePath \"\""
Jul 06 20:27:10.224192 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.224178    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbbrc\" (UniqueName: \"kubernetes.io/projected/9155b9e0-f958-4865-a2a2-6efdaa7e9f2f-kube-api-access-pbbrc\") on node \"ip-10-0-141-123.ec2.internal\" DevicePath \"\""
Jul 06 20:27:10.828856 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.828828    2563 generic.go:356] "Generic (PLEG): container finished" podID="9155b9e0-f958-4865-a2a2-6efdaa7e9f2f" containerID="ff4662e38b7e5900fd751163f5f7031720db100f1155fcec3c19768e35f9f943" exitCode=0
Jul 06 20:27:10.828856 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.828857    2563 generic.go:356] "Generic (PLEG): container finished" podID="9155b9e0-f958-4865-a2a2-6efdaa7e9f2f" containerID="78fa493fc6fe6ca1ad69b58cad660058f3adced525935dcd27c04298a0311624" exitCode=0
Jul 06 20:27:10.829037 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.828867    2563 generic.go:356] "Generic (PLEG): container finished" podID="9155b9e0-f958-4865-a2a2-6efdaa7e9f2f" containerID="9603b18d090f1d1fe2b6df68bd9bff58ec10cf2b479a1f85d3717ee0ab753528" exitCode=0
Jul 06 20:27:10.829037 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.828874    2563 generic.go:356] "Generic (PLEG): container finished" podID="9155b9e0-f958-4865-a2a2-6efdaa7e9f2f" containerID="d4035acf96535ff0faadef756b038c190878f8cc72b2f48b4d2ff1875d417bca" exitCode=0
Jul 06 20:27:10.829037 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.828881    2563 generic.go:356] "Generic (PLEG): container finished" podID="9155b9e0-f958-4865-a2a2-6efdaa7e9f2f" containerID="83ee291b1bd70cf6772f865391715d0bd51b3028744df901141d228d5207897a" exitCode=0
Jul 06 20:27:10.829037 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.828888    2563 generic.go:356] "Generic (PLEG): container finished" podID="9155b9e0-f958-4865-a2a2-6efdaa7e9f2f" containerID="68932e4c162dfa05e9660d65170435619d15d2958f0a09451439be186d3e659b" exitCode=0
Jul 06 20:27:10.829037 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.828988    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:27:10.834125 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.834102    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9155b9e0-f958-4865-a2a2-6efdaa7e9f2f","Type":"ContainerDied","Data":"ff4662e38b7e5900fd751163f5f7031720db100f1155fcec3c19768e35f9f943"}
Jul 06 20:27:10.834245 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.834170    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9155b9e0-f958-4865-a2a2-6efdaa7e9f2f","Type":"ContainerDied","Data":"78fa493fc6fe6ca1ad69b58cad660058f3adced525935dcd27c04298a0311624"}
Jul 06 20:27:10.834245 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.834198    2563 scope.go:117] "RemoveContainer" containerID="ff4662e38b7e5900fd751163f5f7031720db100f1155fcec3c19768e35f9f943"
Jul 06 20:27:10.834245 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.834214    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9155b9e0-f958-4865-a2a2-6efdaa7e9f2f","Type":"ContainerDied","Data":"9603b18d090f1d1fe2b6df68bd9bff58ec10cf2b479a1f85d3717ee0ab753528"}
Jul 06 20:27:10.834410 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.834258    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9155b9e0-f958-4865-a2a2-6efdaa7e9f2f","Type":"ContainerDied","Data":"d4035acf96535ff0faadef756b038c190878f8cc72b2f48b4d2ff1875d417bca"}
Jul 06 20:27:10.834410 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.834298    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9155b9e0-f958-4865-a2a2-6efdaa7e9f2f","Type":"ContainerDied","Data":"83ee291b1bd70cf6772f865391715d0bd51b3028744df901141d228d5207897a"}
Jul 06 20:27:10.834410 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.834340    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9155b9e0-f958-4865-a2a2-6efdaa7e9f2f","Type":"ContainerDied","Data":"68932e4c162dfa05e9660d65170435619d15d2958f0a09451439be186d3e659b"}
Jul 06 20:27:10.834410 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.834380    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9155b9e0-f958-4865-a2a2-6efdaa7e9f2f","Type":"ContainerDied","Data":"48a6339b21f9ab4e8c2ae9e109dd35955312f6c2395d7dd3702ba7a5017e9b2c"}
Jul 06 20:27:10.843717 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.843702    2563 scope.go:117] "RemoveContainer" containerID="78fa493fc6fe6ca1ad69b58cad660058f3adced525935dcd27c04298a0311624"
Jul 06 20:27:10.850508 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.850491    2563 scope.go:117] "RemoveContainer" containerID="9603b18d090f1d1fe2b6df68bd9bff58ec10cf2b479a1f85d3717ee0ab753528"
Jul 06 20:27:10.855301 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.855280    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 06 20:27:10.857504 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.857486    2563 scope.go:117] "RemoveContainer" containerID="d4035acf96535ff0faadef756b038c190878f8cc72b2f48b4d2ff1875d417bca"
Jul 06 20:27:10.860078 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.860060    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 06 20:27:10.864309 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.864294    2563 scope.go:117] "RemoveContainer" containerID="83ee291b1bd70cf6772f865391715d0bd51b3028744df901141d228d5207897a"
Jul 06 20:27:10.870843 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.870824    2563 scope.go:117] "RemoveContainer" containerID="68932e4c162dfa05e9660d65170435619d15d2958f0a09451439be186d3e659b"
Jul 06 20:27:10.877598 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.877581    2563 scope.go:117] "RemoveContainer" containerID="ea17832b610f65b0a5f658cee6415de6566cd26f82b18903aaccc259587a0c7d"
Jul 06 20:27:10.883983 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.883964    2563 scope.go:117] "RemoveContainer" containerID="ff4662e38b7e5900fd751163f5f7031720db100f1155fcec3c19768e35f9f943"
Jul 06 20:27:10.884224 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:27:10.884209    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff4662e38b7e5900fd751163f5f7031720db100f1155fcec3c19768e35f9f943\": container with ID starting with ff4662e38b7e5900fd751163f5f7031720db100f1155fcec3c19768e35f9f943 not found: ID does not exist" containerID="ff4662e38b7e5900fd751163f5f7031720db100f1155fcec3c19768e35f9f943"
Jul 06 20:27:10.884269 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.884232    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff4662e38b7e5900fd751163f5f7031720db100f1155fcec3c19768e35f9f943"} err="failed to get container status \"ff4662e38b7e5900fd751163f5f7031720db100f1155fcec3c19768e35f9f943\": rpc error: code = NotFound desc = could not find container \"ff4662e38b7e5900fd751163f5f7031720db100f1155fcec3c19768e35f9f943\": container with ID starting with ff4662e38b7e5900fd751163f5f7031720db100f1155fcec3c19768e35f9f943 not found: ID does not exist"
Jul 06 20:27:10.884269 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.884249    2563 scope.go:117] "RemoveContainer" containerID="78fa493fc6fe6ca1ad69b58cad660058f3adced525935dcd27c04298a0311624"
Jul 06 20:27:10.884441 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:27:10.884427    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78fa493fc6fe6ca1ad69b58cad660058f3adced525935dcd27c04298a0311624\": container with ID starting with 78fa493fc6fe6ca1ad69b58cad660058f3adced525935dcd27c04298a0311624 not found: ID does not exist" containerID="78fa493fc6fe6ca1ad69b58cad660058f3adced525935dcd27c04298a0311624"
Jul 06 20:27:10.884485 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.884445    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78fa493fc6fe6ca1ad69b58cad660058f3adced525935dcd27c04298a0311624"} err="failed to get container status \"78fa493fc6fe6ca1ad69b58cad660058f3adced525935dcd27c04298a0311624\": rpc error: code = NotFound desc = could not find container \"78fa493fc6fe6ca1ad69b58cad660058f3adced525935dcd27c04298a0311624\": container with ID starting with 78fa493fc6fe6ca1ad69b58cad660058f3adced525935dcd27c04298a0311624 not found: ID does not exist"
Jul 06 20:27:10.884485 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.884456    2563 scope.go:117] "RemoveContainer" containerID="9603b18d090f1d1fe2b6df68bd9bff58ec10cf2b479a1f85d3717ee0ab753528"
Jul 06 20:27:10.884691 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:27:10.884675    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9603b18d090f1d1fe2b6df68bd9bff58ec10cf2b479a1f85d3717ee0ab753528\": container with ID starting with 9603b18d090f1d1fe2b6df68bd9bff58ec10cf2b479a1f85d3717ee0ab753528 not found: ID does not exist" containerID="9603b18d090f1d1fe2b6df68bd9bff58ec10cf2b479a1f85d3717ee0ab753528"
Jul 06 20:27:10.884756 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.884693    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9603b18d090f1d1fe2b6df68bd9bff58ec10cf2b479a1f85d3717ee0ab753528"} err="failed to get container status \"9603b18d090f1d1fe2b6df68bd9bff58ec10cf2b479a1f85d3717ee0ab753528\": rpc error: code = NotFound desc = could not find container \"9603b18d090f1d1fe2b6df68bd9bff58ec10cf2b479a1f85d3717ee0ab753528\": container with ID starting with 9603b18d090f1d1fe2b6df68bd9bff58ec10cf2b479a1f85d3717ee0ab753528 not found: ID does not exist"
Jul 06 20:27:10.884756 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.884706    2563 scope.go:117] "RemoveContainer" containerID="d4035acf96535ff0faadef756b038c190878f8cc72b2f48b4d2ff1875d417bca"
Jul 06 20:27:10.884930 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:27:10.884915    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4035acf96535ff0faadef756b038c190878f8cc72b2f48b4d2ff1875d417bca\": container with ID starting with d4035acf96535ff0faadef756b038c190878f8cc72b2f48b4d2ff1875d417bca not found: ID does not exist" containerID="d4035acf96535ff0faadef756b038c190878f8cc72b2f48b4d2ff1875d417bca"
Jul 06 20:27:10.884975 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.884934    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4035acf96535ff0faadef756b038c190878f8cc72b2f48b4d2ff1875d417bca"} err="failed to get container status \"d4035acf96535ff0faadef756b038c190878f8cc72b2f48b4d2ff1875d417bca\": rpc error: code = NotFound desc = could not find container \"d4035acf96535ff0faadef756b038c190878f8cc72b2f48b4d2ff1875d417bca\": container with ID starting with d4035acf96535ff0faadef756b038c190878f8cc72b2f48b4d2ff1875d417bca not found: ID does not exist"
Jul 06 20:27:10.884975 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.884945    2563 scope.go:117] "RemoveContainer" containerID="83ee291b1bd70cf6772f865391715d0bd51b3028744df901141d228d5207897a"
Jul 06 20:27:10.885134 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:27:10.885121    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83ee291b1bd70cf6772f865391715d0bd51b3028744df901141d228d5207897a\": container with ID starting with 83ee291b1bd70cf6772f865391715d0bd51b3028744df901141d228d5207897a not found: ID does not exist" containerID="83ee291b1bd70cf6772f865391715d0bd51b3028744df901141d228d5207897a"
Jul 06 20:27:10.885168 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.885137    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83ee291b1bd70cf6772f865391715d0bd51b3028744df901141d228d5207897a"} err="failed to get container status \"83ee291b1bd70cf6772f865391715d0bd51b3028744df901141d228d5207897a\": rpc error: code = NotFound desc = could not find container \"83ee291b1bd70cf6772f865391715d0bd51b3028744df901141d228d5207897a\": container with ID starting with 83ee291b1bd70cf6772f865391715d0bd51b3028744df901141d228d5207897a not found: ID does not exist"
Jul 06 20:27:10.885168 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.885148    2563 scope.go:117] "RemoveContainer" containerID="68932e4c162dfa05e9660d65170435619d15d2958f0a09451439be186d3e659b"
Jul 06 20:27:10.885368 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:27:10.885349    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68932e4c162dfa05e9660d65170435619d15d2958f0a09451439be186d3e659b\": container with ID starting with 68932e4c162dfa05e9660d65170435619d15d2958f0a09451439be186d3e659b not found: ID does not exist" containerID="68932e4c162dfa05e9660d65170435619d15d2958f0a09451439be186d3e659b"
Jul 06 20:27:10.885407 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.885373    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68932e4c162dfa05e9660d65170435619d15d2958f0a09451439be186d3e659b"} err="failed to get container status \"68932e4c162dfa05e9660d65170435619d15d2958f0a09451439be186d3e659b\": rpc error: code = NotFound desc = could not find container \"68932e4c162dfa05e9660d65170435619d15d2958f0a09451439be186d3e659b\": container with ID starting with 68932e4c162dfa05e9660d65170435619d15d2958f0a09451439be186d3e659b not found: ID does not exist"
Jul 06 20:27:10.885407 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.885388    2563 scope.go:117] "RemoveContainer" containerID="ea17832b610f65b0a5f658cee6415de6566cd26f82b18903aaccc259587a0c7d"
Jul 06 20:27:10.885592 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:27:10.885577    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea17832b610f65b0a5f658cee6415de6566cd26f82b18903aaccc259587a0c7d\": container with ID starting with ea17832b610f65b0a5f658cee6415de6566cd26f82b18903aaccc259587a0c7d not found: ID does not exist" containerID="ea17832b610f65b0a5f658cee6415de6566cd26f82b18903aaccc259587a0c7d"
Jul 06 20:27:10.885631 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.885595    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea17832b610f65b0a5f658cee6415de6566cd26f82b18903aaccc259587a0c7d"} err="failed to get container status \"ea17832b610f65b0a5f658cee6415de6566cd26f82b18903aaccc259587a0c7d\": rpc error: code = NotFound desc = could not find container \"ea17832b610f65b0a5f658cee6415de6566cd26f82b18903aaccc259587a0c7d\": container with ID starting with ea17832b610f65b0a5f658cee6415de6566cd26f82b18903aaccc259587a0c7d not found: ID does not exist"
Jul 06 20:27:10.885631 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.885609    2563 scope.go:117] "RemoveContainer" containerID="ff4662e38b7e5900fd751163f5f7031720db100f1155fcec3c19768e35f9f943"
Jul 06 20:27:10.885851 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.885832    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff4662e38b7e5900fd751163f5f7031720db100f1155fcec3c19768e35f9f943"} err="failed to get container status \"ff4662e38b7e5900fd751163f5f7031720db100f1155fcec3c19768e35f9f943\": rpc error: code = NotFound desc = could not find container \"ff4662e38b7e5900fd751163f5f7031720db100f1155fcec3c19768e35f9f943\": container with ID starting with ff4662e38b7e5900fd751163f5f7031720db100f1155fcec3c19768e35f9f943 not found: ID does not exist"
Jul 06 20:27:10.885907 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.885852    2563 scope.go:117] "RemoveContainer" containerID="78fa493fc6fe6ca1ad69b58cad660058f3adced525935dcd27c04298a0311624"
Jul 06 20:27:10.886068 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.886051    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78fa493fc6fe6ca1ad69b58cad660058f3adced525935dcd27c04298a0311624"} err="failed to get container status \"78fa493fc6fe6ca1ad69b58cad660058f3adced525935dcd27c04298a0311624\": rpc error: code = NotFound desc = could not find container \"78fa493fc6fe6ca1ad69b58cad660058f3adced525935dcd27c04298a0311624\": container with ID starting with 78fa493fc6fe6ca1ad69b58cad660058f3adced525935dcd27c04298a0311624 not found: ID does not exist"
Jul 06 20:27:10.886118 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.886068    2563 scope.go:117] "RemoveContainer" containerID="9603b18d090f1d1fe2b6df68bd9bff58ec10cf2b479a1f85d3717ee0ab753528"
Jul 06 20:27:10.886308 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.886293    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9603b18d090f1d1fe2b6df68bd9bff58ec10cf2b479a1f85d3717ee0ab753528"} err="failed to get container status \"9603b18d090f1d1fe2b6df68bd9bff58ec10cf2b479a1f85d3717ee0ab753528\": rpc error: code = NotFound desc = could not find container \"9603b18d090f1d1fe2b6df68bd9bff58ec10cf2b479a1f85d3717ee0ab753528\": container with ID starting with 9603b18d090f1d1fe2b6df68bd9bff58ec10cf2b479a1f85d3717ee0ab753528 not found: ID does not exist"
Jul 06 20:27:10.886308 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.886308    2563 scope.go:117] "RemoveContainer" containerID="d4035acf96535ff0faadef756b038c190878f8cc72b2f48b4d2ff1875d417bca"
Jul 06 20:27:10.886488 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.886471    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4035acf96535ff0faadef756b038c190878f8cc72b2f48b4d2ff1875d417bca"} err="failed to get container status \"d4035acf96535ff0faadef756b038c190878f8cc72b2f48b4d2ff1875d417bca\": rpc error: code = NotFound desc = could not find container \"d4035acf96535ff0faadef756b038c190878f8cc72b2f48b4d2ff1875d417bca\": container with ID starting with d4035acf96535ff0faadef756b038c190878f8cc72b2f48b4d2ff1875d417bca not found: ID does not exist"
Jul 06 20:27:10.886531 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.886487    2563 scope.go:117] "RemoveContainer" containerID="83ee291b1bd70cf6772f865391715d0bd51b3028744df901141d228d5207897a"
Jul 06 20:27:10.886659 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.886643    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83ee291b1bd70cf6772f865391715d0bd51b3028744df901141d228d5207897a"} err="failed to get container status \"83ee291b1bd70cf6772f865391715d0bd51b3028744df901141d228d5207897a\": rpc error: code = NotFound desc = could not find container \"83ee291b1bd70cf6772f865391715d0bd51b3028744df901141d228d5207897a\": container with ID starting with 83ee291b1bd70cf6772f865391715d0bd51b3028744df901141d228d5207897a not found: ID does not exist"
Jul 06 20:27:10.886699 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.886661    2563 scope.go:117] "RemoveContainer" containerID="68932e4c162dfa05e9660d65170435619d15d2958f0a09451439be186d3e659b"
Jul 06 20:27:10.886869 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.886853    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68932e4c162dfa05e9660d65170435619d15d2958f0a09451439be186d3e659b"} err="failed to get container status \"68932e4c162dfa05e9660d65170435619d15d2958f0a09451439be186d3e659b\": rpc error: code = NotFound desc = could not find container \"68932e4c162dfa05e9660d65170435619d15d2958f0a09451439be186d3e659b\": container with ID starting with 68932e4c162dfa05e9660d65170435619d15d2958f0a09451439be186d3e659b not found: ID does not exist"
Jul 06 20:27:10.886965 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.886869    2563 scope.go:117] "RemoveContainer" containerID="ea17832b610f65b0a5f658cee6415de6566cd26f82b18903aaccc259587a0c7d"
Jul 06 20:27:10.887091 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.887071    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea17832b610f65b0a5f658cee6415de6566cd26f82b18903aaccc259587a0c7d"} err="failed to get container status \"ea17832b610f65b0a5f658cee6415de6566cd26f82b18903aaccc259587a0c7d\": rpc error: code = NotFound desc = could not find container \"ea17832b610f65b0a5f658cee6415de6566cd26f82b18903aaccc259587a0c7d\": container with ID starting with ea17832b610f65b0a5f658cee6415de6566cd26f82b18903aaccc259587a0c7d not found: ID does not exist"
Jul 06 20:27:10.887138 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.887093    2563 scope.go:117] "RemoveContainer" containerID="ff4662e38b7e5900fd751163f5f7031720db100f1155fcec3c19768e35f9f943"
Jul 06 20:27:10.887424 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.887320    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff4662e38b7e5900fd751163f5f7031720db100f1155fcec3c19768e35f9f943"} err="failed to get container status \"ff4662e38b7e5900fd751163f5f7031720db100f1155fcec3c19768e35f9f943\": rpc error: code = NotFound desc = could not find container \"ff4662e38b7e5900fd751163f5f7031720db100f1155fcec3c19768e35f9f943\": container with ID starting with ff4662e38b7e5900fd751163f5f7031720db100f1155fcec3c19768e35f9f943 not found: ID does not exist"
Jul 06 20:27:10.887424 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.887349    2563 scope.go:117] "RemoveContainer" containerID="78fa493fc6fe6ca1ad69b58cad660058f3adced525935dcd27c04298a0311624"
Jul 06 20:27:10.887631 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.887603    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78fa493fc6fe6ca1ad69b58cad660058f3adced525935dcd27c04298a0311624"} err="failed to get container status \"78fa493fc6fe6ca1ad69b58cad660058f3adced525935dcd27c04298a0311624\": rpc error: code = NotFound desc = could not find container \"78fa493fc6fe6ca1ad69b58cad660058f3adced525935dcd27c04298a0311624\": container with ID starting with 78fa493fc6fe6ca1ad69b58cad660058f3adced525935dcd27c04298a0311624 not found: ID does not exist"
Jul 06 20:27:10.887695 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.887632    2563 scope.go:117] "RemoveContainer" containerID="9603b18d090f1d1fe2b6df68bd9bff58ec10cf2b479a1f85d3717ee0ab753528"
Jul 06 20:27:10.888095 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.888071    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9603b18d090f1d1fe2b6df68bd9bff58ec10cf2b479a1f85d3717ee0ab753528"} err="failed to get container status \"9603b18d090f1d1fe2b6df68bd9bff58ec10cf2b479a1f85d3717ee0ab753528\": rpc error: code = NotFound desc = could not find container \"9603b18d090f1d1fe2b6df68bd9bff58ec10cf2b479a1f85d3717ee0ab753528\": container with ID starting with 9603b18d090f1d1fe2b6df68bd9bff58ec10cf2b479a1f85d3717ee0ab753528 not found: ID does not exist"
Jul 06 20:27:10.888194 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.888096    2563 scope.go:117] "RemoveContainer" containerID="d4035acf96535ff0faadef756b038c190878f8cc72b2f48b4d2ff1875d417bca"
Jul 06 20:27:10.888314 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.888296    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4035acf96535ff0faadef756b038c190878f8cc72b2f48b4d2ff1875d417bca"} err="failed to get container status \"d4035acf96535ff0faadef756b038c190878f8cc72b2f48b4d2ff1875d417bca\": rpc error: code = NotFound desc = could not find container \"d4035acf96535ff0faadef756b038c190878f8cc72b2f48b4d2ff1875d417bca\": container with ID starting with d4035acf96535ff0faadef756b038c190878f8cc72b2f48b4d2ff1875d417bca not found: ID does not exist"
Jul 06 20:27:10.888355 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.888315    2563 scope.go:117] "RemoveContainer" containerID="83ee291b1bd70cf6772f865391715d0bd51b3028744df901141d228d5207897a"
Jul 06 20:27:10.888550 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.888528    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83ee291b1bd70cf6772f865391715d0bd51b3028744df901141d228d5207897a"} err="failed to get container status \"83ee291b1bd70cf6772f865391715d0bd51b3028744df901141d228d5207897a\": rpc error: code = NotFound desc = could not find container \"83ee291b1bd70cf6772f865391715d0bd51b3028744df901141d228d5207897a\": container with ID starting with 83ee291b1bd70cf6772f865391715d0bd51b3028744df901141d228d5207897a not found: ID does not exist"
Jul 06 20:27:10.888603 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.888553    2563 scope.go:117] "RemoveContainer" containerID="68932e4c162dfa05e9660d65170435619d15d2958f0a09451439be186d3e659b"
Jul 06 20:27:10.888873 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.888852    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68932e4c162dfa05e9660d65170435619d15d2958f0a09451439be186d3e659b"} err="failed to get container status \"68932e4c162dfa05e9660d65170435619d15d2958f0a09451439be186d3e659b\": rpc error: code = NotFound desc = could not find container \"68932e4c162dfa05e9660d65170435619d15d2958f0a09451439be186d3e659b\": container with ID starting with 68932e4c162dfa05e9660d65170435619d15d2958f0a09451439be186d3e659b not found: ID does not exist"
Jul 06 20:27:10.888953 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.888876    2563 scope.go:117] "RemoveContainer" containerID="ea17832b610f65b0a5f658cee6415de6566cd26f82b18903aaccc259587a0c7d"
Jul 06 20:27:10.889120 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.889098    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea17832b610f65b0a5f658cee6415de6566cd26f82b18903aaccc259587a0c7d"} err="failed to get container status \"ea17832b610f65b0a5f658cee6415de6566cd26f82b18903aaccc259587a0c7d\": rpc error: code = NotFound desc = could not find container \"ea17832b610f65b0a5f658cee6415de6566cd26f82b18903aaccc259587a0c7d\": container with ID starting with ea17832b610f65b0a5f658cee6415de6566cd26f82b18903aaccc259587a0c7d not found: ID does not exist"
Jul 06 20:27:10.889185 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.889123    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 06 20:27:10.889225 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.889123    2563 scope.go:117] "RemoveContainer" containerID="ff4662e38b7e5900fd751163f5f7031720db100f1155fcec3c19768e35f9f943"
Jul 06 20:27:10.889408 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.889387    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff4662e38b7e5900fd751163f5f7031720db100f1155fcec3c19768e35f9f943"} err="failed to get container status \"ff4662e38b7e5900fd751163f5f7031720db100f1155fcec3c19768e35f9f943\": rpc error: code = NotFound desc = could not find container \"ff4662e38b7e5900fd751163f5f7031720db100f1155fcec3c19768e35f9f943\": container with ID starting with ff4662e38b7e5900fd751163f5f7031720db100f1155fcec3c19768e35f9f943 not found: ID does not exist"
Jul 06 20:27:10.889408 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.889405    2563 scope.go:117] "RemoveContainer" containerID="78fa493fc6fe6ca1ad69b58cad660058f3adced525935dcd27c04298a0311624"
Jul 06 20:27:10.889480 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.889462    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9155b9e0-f958-4865-a2a2-6efdaa7e9f2f" containerName="kube-rbac-proxy"
Jul 06 20:27:10.889480 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.889478    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="9155b9e0-f958-4865-a2a2-6efdaa7e9f2f" containerName="kube-rbac-proxy"
Jul 06 20:27:10.889535 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.889497    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9155b9e0-f958-4865-a2a2-6efdaa7e9f2f" containerName="config-reloader"
Jul 06 20:27:10.889535 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.889506    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="9155b9e0-f958-4865-a2a2-6efdaa7e9f2f" containerName="config-reloader"
Jul 06 20:27:10.889535 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.889522    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9155b9e0-f958-4865-a2a2-6efdaa7e9f2f" containerName="kube-rbac-proxy-thanos"
Jul 06 20:27:10.889535 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.889530    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="9155b9e0-f958-4865-a2a2-6efdaa7e9f2f" containerName="kube-rbac-proxy-thanos"
Jul 06 20:27:10.889650 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.889541    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9155b9e0-f958-4865-a2a2-6efdaa7e9f2f" containerName="thanos-sidecar"
Jul 06 20:27:10.889650 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.889549    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="9155b9e0-f958-4865-a2a2-6efdaa7e9f2f" containerName="thanos-sidecar"
Jul 06 20:27:10.889650 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.889558    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9155b9e0-f958-4865-a2a2-6efdaa7e9f2f" containerName="kube-rbac-proxy-web"
Jul 06 20:27:10.889650 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.889565    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="9155b9e0-f958-4865-a2a2-6efdaa7e9f2f" containerName="kube-rbac-proxy-web"
Jul 06 20:27:10.889650 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.889582    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9155b9e0-f958-4865-a2a2-6efdaa7e9f2f" containerName="init-config-reloader"
Jul 06 20:27:10.889650 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.889589    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="9155b9e0-f958-4865-a2a2-6efdaa7e9f2f" containerName="init-config-reloader"
Jul 06 20:27:10.889650 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.889601    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9155b9e0-f958-4865-a2a2-6efdaa7e9f2f" containerName="prometheus"
Jul 06 20:27:10.889650 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.889606    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="9155b9e0-f958-4865-a2a2-6efdaa7e9f2f" containerName="prometheus"
Jul 06 20:27:10.889650 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.889639    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78fa493fc6fe6ca1ad69b58cad660058f3adced525935dcd27c04298a0311624"} err="failed to get container status \"78fa493fc6fe6ca1ad69b58cad660058f3adced525935dcd27c04298a0311624\": rpc error: code = NotFound desc = could not find container \"78fa493fc6fe6ca1ad69b58cad660058f3adced525935dcd27c04298a0311624\": container with ID starting with 78fa493fc6fe6ca1ad69b58cad660058f3adced525935dcd27c04298a0311624 not found: ID does not exist"
Jul 06 20:27:10.889986 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.889657    2563 scope.go:117] "RemoveContainer" containerID="9603b18d090f1d1fe2b6df68bd9bff58ec10cf2b479a1f85d3717ee0ab753528"
Jul 06 20:27:10.889986 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.889663    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="9155b9e0-f958-4865-a2a2-6efdaa7e9f2f" containerName="prometheus"
Jul 06 20:27:10.889986 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.889672    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="9155b9e0-f958-4865-a2a2-6efdaa7e9f2f" containerName="config-reloader"
Jul 06 20:27:10.889986 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.889679    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="9155b9e0-f958-4865-a2a2-6efdaa7e9f2f" containerName="kube-rbac-proxy-web"
Jul 06 20:27:10.889986 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.889685    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="9155b9e0-f958-4865-a2a2-6efdaa7e9f2f" containerName="kube-rbac-proxy-thanos"
Jul 06 20:27:10.889986 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.889690    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="9155b9e0-f958-4865-a2a2-6efdaa7e9f2f" containerName="thanos-sidecar"
Jul 06 20:27:10.889986 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.889696    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="9155b9e0-f958-4865-a2a2-6efdaa7e9f2f" containerName="kube-rbac-proxy"
Jul 06 20:27:10.889986 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.889906    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9603b18d090f1d1fe2b6df68bd9bff58ec10cf2b479a1f85d3717ee0ab753528"} err="failed to get container status \"9603b18d090f1d1fe2b6df68bd9bff58ec10cf2b479a1f85d3717ee0ab753528\": rpc error: code = NotFound desc = could not find container \"9603b18d090f1d1fe2b6df68bd9bff58ec10cf2b479a1f85d3717ee0ab753528\": container with ID starting with 9603b18d090f1d1fe2b6df68bd9bff58ec10cf2b479a1f85d3717ee0ab753528 not found: ID does not exist"
Jul 06 20:27:10.889986 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.889919    2563 scope.go:117] "RemoveContainer" containerID="d4035acf96535ff0faadef756b038c190878f8cc72b2f48b4d2ff1875d417bca"
Jul 06 20:27:10.890317 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.890138    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4035acf96535ff0faadef756b038c190878f8cc72b2f48b4d2ff1875d417bca"} err="failed to get container status \"d4035acf96535ff0faadef756b038c190878f8cc72b2f48b4d2ff1875d417bca\": rpc error: code = NotFound desc = could not find container \"d4035acf96535ff0faadef756b038c190878f8cc72b2f48b4d2ff1875d417bca\": container with ID starting with d4035acf96535ff0faadef756b038c190878f8cc72b2f48b4d2ff1875d417bca not found: ID does not exist"
Jul 06 20:27:10.890317 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.890153    2563 scope.go:117] "RemoveContainer" containerID="83ee291b1bd70cf6772f865391715d0bd51b3028744df901141d228d5207897a"
Jul 06 20:27:10.890394 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.890371    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83ee291b1bd70cf6772f865391715d0bd51b3028744df901141d228d5207897a"} err="failed to get container status \"83ee291b1bd70cf6772f865391715d0bd51b3028744df901141d228d5207897a\": rpc error: code = NotFound desc = could not find container \"83ee291b1bd70cf6772f865391715d0bd51b3028744df901141d228d5207897a\": container with ID starting with 83ee291b1bd70cf6772f865391715d0bd51b3028744df901141d228d5207897a not found: ID does not exist"
Jul 06 20:27:10.890432 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.890397    2563 scope.go:117] "RemoveContainer" containerID="68932e4c162dfa05e9660d65170435619d15d2958f0a09451439be186d3e659b"
Jul 06 20:27:10.890722 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.890687    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68932e4c162dfa05e9660d65170435619d15d2958f0a09451439be186d3e659b"} err="failed to get container status \"68932e4c162dfa05e9660d65170435619d15d2958f0a09451439be186d3e659b\": rpc error: code = NotFound desc = could not find container \"68932e4c162dfa05e9660d65170435619d15d2958f0a09451439be186d3e659b\": container with ID starting with 68932e4c162dfa05e9660d65170435619d15d2958f0a09451439be186d3e659b not found: ID does not exist"
Jul 06 20:27:10.890722 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.890712    2563 scope.go:117] "RemoveContainer" containerID="ea17832b610f65b0a5f658cee6415de6566cd26f82b18903aaccc259587a0c7d"
Jul 06 20:27:10.891024 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.891006    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea17832b610f65b0a5f658cee6415de6566cd26f82b18903aaccc259587a0c7d"} err="failed to get container status \"ea17832b610f65b0a5f658cee6415de6566cd26f82b18903aaccc259587a0c7d\": rpc error: code = NotFound desc = could not find container \"ea17832b610f65b0a5f658cee6415de6566cd26f82b18903aaccc259587a0c7d\": container with ID starting with ea17832b610f65b0a5f658cee6415de6566cd26f82b18903aaccc259587a0c7d not found: ID does not exist"
Jul 06 20:27:10.891024 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.891025    2563 scope.go:117] "RemoveContainer" containerID="ff4662e38b7e5900fd751163f5f7031720db100f1155fcec3c19768e35f9f943"
Jul 06 20:27:10.891262 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.891247    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff4662e38b7e5900fd751163f5f7031720db100f1155fcec3c19768e35f9f943"} err="failed to get container status \"ff4662e38b7e5900fd751163f5f7031720db100f1155fcec3c19768e35f9f943\": rpc error: code = NotFound desc = could not find container \"ff4662e38b7e5900fd751163f5f7031720db100f1155fcec3c19768e35f9f943\": container with ID starting with ff4662e38b7e5900fd751163f5f7031720db100f1155fcec3c19768e35f9f943 not found: ID does not exist"
Jul 06 20:27:10.891313 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.891263    2563 scope.go:117] "RemoveContainer" containerID="78fa493fc6fe6ca1ad69b58cad660058f3adced525935dcd27c04298a0311624"
Jul 06 20:27:10.891471 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.891453    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78fa493fc6fe6ca1ad69b58cad660058f3adced525935dcd27c04298a0311624"} err="failed to get container status \"78fa493fc6fe6ca1ad69b58cad660058f3adced525935dcd27c04298a0311624\": rpc error: code = NotFound desc = could not find container \"78fa493fc6fe6ca1ad69b58cad660058f3adced525935dcd27c04298a0311624\": container with ID starting with 78fa493fc6fe6ca1ad69b58cad660058f3adced525935dcd27c04298a0311624 not found: ID does not exist"
Jul 06 20:27:10.891521 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.891472    2563 scope.go:117] "RemoveContainer" containerID="9603b18d090f1d1fe2b6df68bd9bff58ec10cf2b479a1f85d3717ee0ab753528"
Jul 06 20:27:10.891672 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.891656    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9603b18d090f1d1fe2b6df68bd9bff58ec10cf2b479a1f85d3717ee0ab753528"} err="failed to get container status \"9603b18d090f1d1fe2b6df68bd9bff58ec10cf2b479a1f85d3717ee0ab753528\": rpc error: code = NotFound desc = could not find container \"9603b18d090f1d1fe2b6df68bd9bff58ec10cf2b479a1f85d3717ee0ab753528\": container with ID starting with 9603b18d090f1d1fe2b6df68bd9bff58ec10cf2b479a1f85d3717ee0ab753528 not found: ID does not exist"
Jul 06 20:27:10.891718 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.891675    2563 scope.go:117] "RemoveContainer" containerID="d4035acf96535ff0faadef756b038c190878f8cc72b2f48b4d2ff1875d417bca"
Jul 06 20:27:10.892112 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.892085    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4035acf96535ff0faadef756b038c190878f8cc72b2f48b4d2ff1875d417bca"} err="failed to get container status \"d4035acf96535ff0faadef756b038c190878f8cc72b2f48b4d2ff1875d417bca\": rpc error: code = NotFound desc = could not find container \"d4035acf96535ff0faadef756b038c190878f8cc72b2f48b4d2ff1875d417bca\": container with ID starting with d4035acf96535ff0faadef756b038c190878f8cc72b2f48b4d2ff1875d417bca not found: ID does not exist"
Jul 06 20:27:10.892112 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.892109    2563 scope.go:117] "RemoveContainer" containerID="83ee291b1bd70cf6772f865391715d0bd51b3028744df901141d228d5207897a"
Jul 06 20:27:10.892382 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.892349    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83ee291b1bd70cf6772f865391715d0bd51b3028744df901141d228d5207897a"} err="failed to get container status \"83ee291b1bd70cf6772f865391715d0bd51b3028744df901141d228d5207897a\": rpc error: code = NotFound desc = could not find container \"83ee291b1bd70cf6772f865391715d0bd51b3028744df901141d228d5207897a\": container with ID starting with 83ee291b1bd70cf6772f865391715d0bd51b3028744df901141d228d5207897a not found: ID does not exist"
Jul 06 20:27:10.892382 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.892374    2563 scope.go:117] "RemoveContainer" containerID="68932e4c162dfa05e9660d65170435619d15d2958f0a09451439be186d3e659b"
Jul 06 20:27:10.894555 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.892767    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68932e4c162dfa05e9660d65170435619d15d2958f0a09451439be186d3e659b"} err="failed to get container status \"68932e4c162dfa05e9660d65170435619d15d2958f0a09451439be186d3e659b\": rpc error: code = NotFound desc = could not find container \"68932e4c162dfa05e9660d65170435619d15d2958f0a09451439be186d3e659b\": container with ID starting with 68932e4c162dfa05e9660d65170435619d15d2958f0a09451439be186d3e659b not found: ID does not exist"
Jul 06 20:27:10.894555 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.892791    2563 scope.go:117] "RemoveContainer" containerID="ea17832b610f65b0a5f658cee6415de6566cd26f82b18903aaccc259587a0c7d"
Jul 06 20:27:10.894555 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.893052    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea17832b610f65b0a5f658cee6415de6566cd26f82b18903aaccc259587a0c7d"} err="failed to get container status \"ea17832b610f65b0a5f658cee6415de6566cd26f82b18903aaccc259587a0c7d\": rpc error: code = NotFound desc = could not find container \"ea17832b610f65b0a5f658cee6415de6566cd26f82b18903aaccc259587a0c7d\": container with ID starting with ea17832b610f65b0a5f658cee6415de6566cd26f82b18903aaccc259587a0c7d not found: ID does not exist"
Jul 06 20:27:10.894555 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.893082    2563 scope.go:117] "RemoveContainer" containerID="ff4662e38b7e5900fd751163f5f7031720db100f1155fcec3c19768e35f9f943"
Jul 06 20:27:10.894555 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.893314    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff4662e38b7e5900fd751163f5f7031720db100f1155fcec3c19768e35f9f943"} err="failed to get container status \"ff4662e38b7e5900fd751163f5f7031720db100f1155fcec3c19768e35f9f943\": rpc error: code = NotFound desc = could not find container \"ff4662e38b7e5900fd751163f5f7031720db100f1155fcec3c19768e35f9f943\": container with ID starting with ff4662e38b7e5900fd751163f5f7031720db100f1155fcec3c19768e35f9f943 not found: ID does not exist"
Jul 06 20:27:10.894555 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.893343    2563 scope.go:117] "RemoveContainer" containerID="78fa493fc6fe6ca1ad69b58cad660058f3adced525935dcd27c04298a0311624"
Jul 06 20:27:10.894555 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.893563    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78fa493fc6fe6ca1ad69b58cad660058f3adced525935dcd27c04298a0311624"} err="failed to get container status \"78fa493fc6fe6ca1ad69b58cad660058f3adced525935dcd27c04298a0311624\": rpc error: code = NotFound desc = could not find container \"78fa493fc6fe6ca1ad69b58cad660058f3adced525935dcd27c04298a0311624\": container with ID starting with 78fa493fc6fe6ca1ad69b58cad660058f3adced525935dcd27c04298a0311624 not found: ID does not exist"
Jul 06 20:27:10.894555 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.893592    2563 scope.go:117] "RemoveContainer" containerID="9603b18d090f1d1fe2b6df68bd9bff58ec10cf2b479a1f85d3717ee0ab753528"
Jul 06 20:27:10.894555 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.893827    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9603b18d090f1d1fe2b6df68bd9bff58ec10cf2b479a1f85d3717ee0ab753528"} err="failed to get container status \"9603b18d090f1d1fe2b6df68bd9bff58ec10cf2b479a1f85d3717ee0ab753528\": rpc error: code = NotFound desc = could not find container \"9603b18d090f1d1fe2b6df68bd9bff58ec10cf2b479a1f85d3717ee0ab753528\": container with ID starting with 9603b18d090f1d1fe2b6df68bd9bff58ec10cf2b479a1f85d3717ee0ab753528 not found: ID does not exist"
Jul 06 20:27:10.894555 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.893858    2563 scope.go:117] "RemoveContainer" containerID="d4035acf96535ff0faadef756b038c190878f8cc72b2f48b4d2ff1875d417bca"
Jul 06 20:27:10.894555 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.894079    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4035acf96535ff0faadef756b038c190878f8cc72b2f48b4d2ff1875d417bca"} err="failed to get container status \"d4035acf96535ff0faadef756b038c190878f8cc72b2f48b4d2ff1875d417bca\": rpc error: code = NotFound desc = could not find container \"d4035acf96535ff0faadef756b038c190878f8cc72b2f48b4d2ff1875d417bca\": container with ID starting with d4035acf96535ff0faadef756b038c190878f8cc72b2f48b4d2ff1875d417bca not found: ID does not exist"
Jul 06 20:27:10.894555 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.894108    2563 scope.go:117] "RemoveContainer" containerID="83ee291b1bd70cf6772f865391715d0bd51b3028744df901141d228d5207897a"
Jul 06 20:27:10.894555 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.894319    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83ee291b1bd70cf6772f865391715d0bd51b3028744df901141d228d5207897a"} err="failed to get container status \"83ee291b1bd70cf6772f865391715d0bd51b3028744df901141d228d5207897a\": rpc error: code = NotFound desc = could not find container \"83ee291b1bd70cf6772f865391715d0bd51b3028744df901141d228d5207897a\": container with ID starting with 83ee291b1bd70cf6772f865391715d0bd51b3028744df901141d228d5207897a not found: ID does not exist"
Jul 06 20:27:10.894555 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.894347    2563 scope.go:117] "RemoveContainer" containerID="68932e4c162dfa05e9660d65170435619d15d2958f0a09451439be186d3e659b"
Jul 06 20:27:10.895340 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.894565    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68932e4c162dfa05e9660d65170435619d15d2958f0a09451439be186d3e659b"} err="failed to get container status \"68932e4c162dfa05e9660d65170435619d15d2958f0a09451439be186d3e659b\": rpc error: code = NotFound desc = could not find container \"68932e4c162dfa05e9660d65170435619d15d2958f0a09451439be186d3e659b\": container with ID starting with 68932e4c162dfa05e9660d65170435619d15d2958f0a09451439be186d3e659b not found: ID does not exist"
Jul 06 20:27:10.895340 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.894596    2563 scope.go:117] "RemoveContainer" containerID="ea17832b610f65b0a5f658cee6415de6566cd26f82b18903aaccc259587a0c7d"
Jul 06 20:27:10.895340 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.894751    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:27:10.895340 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.894905    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea17832b610f65b0a5f658cee6415de6566cd26f82b18903aaccc259587a0c7d"} err="failed to get container status \"ea17832b610f65b0a5f658cee6415de6566cd26f82b18903aaccc259587a0c7d\": rpc error: code = NotFound desc = could not find container \"ea17832b610f65b0a5f658cee6415de6566cd26f82b18903aaccc259587a0c7d\": container with ID starting with ea17832b610f65b0a5f658cee6415de6566cd26f82b18903aaccc259587a0c7d not found: ID does not exist"
Jul 06 20:27:10.896904 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.896886    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 06 20:27:10.896986 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.896904    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 06 20:27:10.896986 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.896906    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 06 20:27:10.897267 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.897250    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 06 20:27:10.897374 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.897265    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 06 20:27:10.897374 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.897256    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 06 20:27:10.897918 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.897824    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 06 20:27:10.903388 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.900982    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 06 20:27:10.903388 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.901163    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 06 20:27:10.903388 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.901371    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 06 20:27:10.903388 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.901611    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 06 20:27:10.903388 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.902203    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-7826s\""
Jul 06 20:27:10.903388 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.902281    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 06 20:27:10.903388 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.902461    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-fdef1ap23nrbg\""
Jul 06 20:27:10.905712 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.905694    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 06 20:27:10.908076 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.908056    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 06 20:27:10.910145 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.910124    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 06 20:27:10.930296 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.930273    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a62a9219-ae33-42ff-800a-4fd7c8b9c908-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a62a9219-ae33-42ff-800a-4fd7c8b9c908\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:27:10.930392 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.930307    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a62a9219-ae33-42ff-800a-4fd7c8b9c908-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a62a9219-ae33-42ff-800a-4fd7c8b9c908\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:27:10.930392 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.930338    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a62a9219-ae33-42ff-800a-4fd7c8b9c908-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a62a9219-ae33-42ff-800a-4fd7c8b9c908\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:27:10.930464 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.930401    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7z78\" (UniqueName: \"kubernetes.io/projected/a62a9219-ae33-42ff-800a-4fd7c8b9c908-kube-api-access-c7z78\") pod \"prometheus-k8s-0\" (UID: \"a62a9219-ae33-42ff-800a-4fd7c8b9c908\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:27:10.930464 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.930450    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a62a9219-ae33-42ff-800a-4fd7c8b9c908-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a62a9219-ae33-42ff-800a-4fd7c8b9c908\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:27:10.930522 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.930487    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a62a9219-ae33-42ff-800a-4fd7c8b9c908-web-config\") pod \"prometheus-k8s-0\" (UID: \"a62a9219-ae33-42ff-800a-4fd7c8b9c908\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:27:10.930522 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.930509    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a62a9219-ae33-42ff-800a-4fd7c8b9c908-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a62a9219-ae33-42ff-800a-4fd7c8b9c908\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:27:10.930624 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.930552    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a62a9219-ae33-42ff-800a-4fd7c8b9c908-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a62a9219-ae33-42ff-800a-4fd7c8b9c908\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:27:10.930624 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.930608    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a62a9219-ae33-42ff-800a-4fd7c8b9c908-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a62a9219-ae33-42ff-800a-4fd7c8b9c908\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:27:10.930709 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.930639    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a62a9219-ae33-42ff-800a-4fd7c8b9c908-config\") pod \"prometheus-k8s-0\" (UID: \"a62a9219-ae33-42ff-800a-4fd7c8b9c908\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:27:10.930709 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.930661    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a62a9219-ae33-42ff-800a-4fd7c8b9c908-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a62a9219-ae33-42ff-800a-4fd7c8b9c908\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:27:10.930709 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.930687    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a62a9219-ae33-42ff-800a-4fd7c8b9c908-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a62a9219-ae33-42ff-800a-4fd7c8b9c908\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:27:10.930840 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.930752    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a62a9219-ae33-42ff-800a-4fd7c8b9c908-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a62a9219-ae33-42ff-800a-4fd7c8b9c908\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:27:10.930840 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.930769    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a62a9219-ae33-42ff-800a-4fd7c8b9c908-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a62a9219-ae33-42ff-800a-4fd7c8b9c908\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:27:10.930840 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.930793    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a62a9219-ae33-42ff-800a-4fd7c8b9c908-config-out\") pod \"prometheus-k8s-0\" (UID: \"a62a9219-ae33-42ff-800a-4fd7c8b9c908\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:27:10.930840 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.930833    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a62a9219-ae33-42ff-800a-4fd7c8b9c908-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a62a9219-ae33-42ff-800a-4fd7c8b9c908\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:27:10.930973 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.930854    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a62a9219-ae33-42ff-800a-4fd7c8b9c908-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a62a9219-ae33-42ff-800a-4fd7c8b9c908\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:27:10.930973 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.930879    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a62a9219-ae33-42ff-800a-4fd7c8b9c908-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a62a9219-ae33-42ff-800a-4fd7c8b9c908\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:27:10.930973 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.930912    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a62a9219-ae33-42ff-800a-4fd7c8b9c908-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a62a9219-ae33-42ff-800a-4fd7c8b9c908\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:27:10.930973 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:10.930927    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a62a9219-ae33-42ff-800a-4fd7c8b9c908-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a62a9219-ae33-42ff-800a-4fd7c8b9c908\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:27:11.031698 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:11.031672    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a62a9219-ae33-42ff-800a-4fd7c8b9c908-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a62a9219-ae33-42ff-800a-4fd7c8b9c908\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:27:11.031698 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:11.031701    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a62a9219-ae33-42ff-800a-4fd7c8b9c908-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a62a9219-ae33-42ff-800a-4fd7c8b9c908\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:27:11.031933 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:11.031720    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a62a9219-ae33-42ff-800a-4fd7c8b9c908-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a62a9219-ae33-42ff-800a-4fd7c8b9c908\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:27:11.031933 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:11.031752    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a62a9219-ae33-42ff-800a-4fd7c8b9c908-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a62a9219-ae33-42ff-800a-4fd7c8b9c908\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:27:11.031933 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:11.031874    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a62a9219-ae33-42ff-800a-4fd7c8b9c908-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a62a9219-ae33-42ff-800a-4fd7c8b9c908\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:27:11.031933 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:11.031915    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7z78\" (UniqueName: \"kubernetes.io/projected/a62a9219-ae33-42ff-800a-4fd7c8b9c908-kube-api-access-c7z78\") pod \"prometheus-k8s-0\" (UID: \"a62a9219-ae33-42ff-800a-4fd7c8b9c908\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:27:11.032134 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:11.031935    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a62a9219-ae33-42ff-800a-4fd7c8b9c908-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a62a9219-ae33-42ff-800a-4fd7c8b9c908\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:27:11.032134 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:11.031984    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a62a9219-ae33-42ff-800a-4fd7c8b9c908-web-config\") pod \"prometheus-k8s-0\" (UID: \"a62a9219-ae33-42ff-800a-4fd7c8b9c908\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:27:11.032134 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:11.032014    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a62a9219-ae33-42ff-800a-4fd7c8b9c908-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a62a9219-ae33-42ff-800a-4fd7c8b9c908\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:27:11.032134 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:11.032045    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a62a9219-ae33-42ff-800a-4fd7c8b9c908-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a62a9219-ae33-42ff-800a-4fd7c8b9c908\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:27:11.032134 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:11.032077    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a62a9219-ae33-42ff-800a-4fd7c8b9c908-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a62a9219-ae33-42ff-800a-4fd7c8b9c908\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:27:11.032134 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:11.032111    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a62a9219-ae33-42ff-800a-4fd7c8b9c908-config\") pod \"prometheus-k8s-0\" (UID: \"a62a9219-ae33-42ff-800a-4fd7c8b9c908\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:27:11.032134 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:11.032132    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a62a9219-ae33-42ff-800a-4fd7c8b9c908-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a62a9219-ae33-42ff-800a-4fd7c8b9c908\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:27:11.032524 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:11.032155    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a62a9219-ae33-42ff-800a-4fd7c8b9c908-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a62a9219-ae33-42ff-800a-4fd7c8b9c908\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:27:11.032524 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:11.032180    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a62a9219-ae33-42ff-800a-4fd7c8b9c908-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a62a9219-ae33-42ff-800a-4fd7c8b9c908\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:27:11.032524 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:11.032202    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a62a9219-ae33-42ff-800a-4fd7c8b9c908-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a62a9219-ae33-42ff-800a-4fd7c8b9c908\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:27:11.032524 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:11.032249    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a62a9219-ae33-42ff-800a-4fd7c8b9c908-config-out\") pod \"prometheus-k8s-0\" (UID: \"a62a9219-ae33-42ff-800a-4fd7c8b9c908\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:27:11.032524 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:11.032279    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a62a9219-ae33-42ff-800a-4fd7c8b9c908-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a62a9219-ae33-42ff-800a-4fd7c8b9c908\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:27:11.032524 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:11.032310    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a62a9219-ae33-42ff-800a-4fd7c8b9c908-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a62a9219-ae33-42ff-800a-4fd7c8b9c908\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:27:11.032524 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:11.032339    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a62a9219-ae33-42ff-800a-4fd7c8b9c908-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a62a9219-ae33-42ff-800a-4fd7c8b9c908\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:27:11.032524 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:11.032490    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a62a9219-ae33-42ff-800a-4fd7c8b9c908-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a62a9219-ae33-42ff-800a-4fd7c8b9c908\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:27:11.032949 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:11.032910    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a62a9219-ae33-42ff-800a-4fd7c8b9c908-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a62a9219-ae33-42ff-800a-4fd7c8b9c908\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:27:11.033020 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:11.032997    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a62a9219-ae33-42ff-800a-4fd7c8b9c908-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a62a9219-ae33-42ff-800a-4fd7c8b9c908\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:27:11.033455 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:11.033430    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a62a9219-ae33-42ff-800a-4fd7c8b9c908-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a62a9219-ae33-42ff-800a-4fd7c8b9c908\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:27:11.034985 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:11.034958    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a62a9219-ae33-42ff-800a-4fd7c8b9c908-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a62a9219-ae33-42ff-800a-4fd7c8b9c908\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:27:11.035098 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:11.035004    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a62a9219-ae33-42ff-800a-4fd7c8b9c908-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a62a9219-ae33-42ff-800a-4fd7c8b9c908\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:27:11.035098 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:11.035029    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a62a9219-ae33-42ff-800a-4fd7c8b9c908-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a62a9219-ae33-42ff-800a-4fd7c8b9c908\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:27:11.035643 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:11.035618    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a62a9219-ae33-42ff-800a-4fd7c8b9c908-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a62a9219-ae33-42ff-800a-4fd7c8b9c908\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:27:11.035763 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:11.035645    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a62a9219-ae33-42ff-800a-4fd7c8b9c908-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a62a9219-ae33-42ff-800a-4fd7c8b9c908\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:27:11.035889 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:11.035867    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a62a9219-ae33-42ff-800a-4fd7c8b9c908-config-out\") pod \"prometheus-k8s-0\" (UID: \"a62a9219-ae33-42ff-800a-4fd7c8b9c908\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:27:11.036501 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:11.036162    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a62a9219-ae33-42ff-800a-4fd7c8b9c908-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a62a9219-ae33-42ff-800a-4fd7c8b9c908\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:27:11.036501 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:11.036393    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a62a9219-ae33-42ff-800a-4fd7c8b9c908-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a62a9219-ae33-42ff-800a-4fd7c8b9c908\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:27:11.036666 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:11.036603    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a62a9219-ae33-42ff-800a-4fd7c8b9c908-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a62a9219-ae33-42ff-800a-4fd7c8b9c908\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:27:11.036666 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:11.036624    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a62a9219-ae33-42ff-800a-4fd7c8b9c908-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a62a9219-ae33-42ff-800a-4fd7c8b9c908\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:27:11.036868 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:11.036841    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a62a9219-ae33-42ff-800a-4fd7c8b9c908-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a62a9219-ae33-42ff-800a-4fd7c8b9c908\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:27:11.037175 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:11.037154    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/a62a9219-ae33-42ff-800a-4fd7c8b9c908-config\") pod \"prometheus-k8s-0\" (UID: \"a62a9219-ae33-42ff-800a-4fd7c8b9c908\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:27:11.037400 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:11.037385    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a62a9219-ae33-42ff-800a-4fd7c8b9c908-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a62a9219-ae33-42ff-800a-4fd7c8b9c908\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:27:11.038352 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:11.038336    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a62a9219-ae33-42ff-800a-4fd7c8b9c908-web-config\") pod \"prometheus-k8s-0\" (UID: \"a62a9219-ae33-42ff-800a-4fd7c8b9c908\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:27:11.038646 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:11.038630    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a62a9219-ae33-42ff-800a-4fd7c8b9c908-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a62a9219-ae33-42ff-800a-4fd7c8b9c908\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:27:11.042094 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:11.042075    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7z78\" (UniqueName: \"kubernetes.io/projected/a62a9219-ae33-42ff-800a-4fd7c8b9c908-kube-api-access-c7z78\") pod \"prometheus-k8s-0\" (UID: \"a62a9219-ae33-42ff-800a-4fd7c8b9c908\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:27:11.211373 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:11.211260    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:27:11.349007 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:11.348947    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 06 20:27:11.350999 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:27:11.350977    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda62a9219_ae33_42ff_800a_4fd7c8b9c908.slice/crio-3d1a8871231f4fe029d689ace1b277acdaf324627ced82d239b82fd7493f14fa WatchSource:0}: Error finding container 3d1a8871231f4fe029d689ace1b277acdaf324627ced82d239b82fd7493f14fa: Status 404 returned error can't find the container with id 3d1a8871231f4fe029d689ace1b277acdaf324627ced82d239b82fd7493f14fa
Jul 06 20:27:11.835012 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:11.834979    2563 generic.go:356] "Generic (PLEG): container finished" podID="a62a9219-ae33-42ff-800a-4fd7c8b9c908" containerID="ec247c6200907213cb8b45c505631adced1559cf8f38384484d6bbd9698a7940" exitCode=0
Jul 06 20:27:11.835164 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:11.835067    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a62a9219-ae33-42ff-800a-4fd7c8b9c908","Type":"ContainerDied","Data":"ec247c6200907213cb8b45c505631adced1559cf8f38384484d6bbd9698a7940"}
Jul 06 20:27:11.835164 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:11.835118    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a62a9219-ae33-42ff-800a-4fd7c8b9c908","Type":"ContainerStarted","Data":"3d1a8871231f4fe029d689ace1b277acdaf324627ced82d239b82fd7493f14fa"}
Jul 06 20:27:12.837060 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:12.834859    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9155b9e0-f958-4865-a2a2-6efdaa7e9f2f" path="/var/lib/kubelet/pods/9155b9e0-f958-4865-a2a2-6efdaa7e9f2f/volumes"
Jul 06 20:27:12.844137 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:12.844109    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a62a9219-ae33-42ff-800a-4fd7c8b9c908","Type":"ContainerStarted","Data":"a279e52a736124000dd9549b68592105fe4813851add113199f4f7473960ef4a"}
Jul 06 20:27:12.844281 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:12.844174    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a62a9219-ae33-42ff-800a-4fd7c8b9c908","Type":"ContainerStarted","Data":"c7b42f579038fe4bb492304b1037f7864aae4d92a0d9e90ef7b771f2f864418c"}
Jul 06 20:27:12.844281 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:12.844218    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a62a9219-ae33-42ff-800a-4fd7c8b9c908","Type":"ContainerStarted","Data":"9ab229e7ccbcc140ccc7053d5c40d1a0d7aabe45eb4aecc7932771dad5546046"}
Jul 06 20:27:12.844281 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:12.844254    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a62a9219-ae33-42ff-800a-4fd7c8b9c908","Type":"ContainerStarted","Data":"952f60ccc9864c2674d8578c50d46c18d0c9e9a775b92591093e1468a05794ad"}
Jul 06 20:27:12.844445 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:12.844288    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a62a9219-ae33-42ff-800a-4fd7c8b9c908","Type":"ContainerStarted","Data":"4ff50eae1a31108385cc6efd8adefa7e3144c8f24f42aa6337b885edd42d2a45"}
Jul 06 20:27:12.844445 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:12.844327    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a62a9219-ae33-42ff-800a-4fd7c8b9c908","Type":"ContainerStarted","Data":"20cdf4670ff969d22217346921d93b74b8b83007f91a807be153601a91c83511"}
Jul 06 20:27:12.873882 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:12.873801    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.854855353 podStartE2EDuration="2.873783529s" totalImagesPullingTime="0s" totalInitContainerRuntime="18.928176ms" isStatefulPod=false podCreationTimestamp="2026-07-06 20:27:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:27:12.872626504 +0000 UTC m=+196.588737166" watchObservedRunningTime="2026-07-06 20:27:12.873783529 +0000 UTC m=+196.589894191"
Jul 06 20:27:15.367794 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:15.367760    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:27:15.463354 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:15.463316    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:27:15.525436 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:15.525404    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:27:16.089211 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:16.089166    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:27:16.189698 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:16.189669    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:27:16.212147 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:16.212116    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:27:18.538522 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:18.538485    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:27:26.208151 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:26.208116    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-78b5766db6-4r7qh"]
Jul 06 20:27:26.328069 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:26.328022    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:27:26.428217 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:26.428170    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:27:26.523455 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:26.523367    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:27:45.445292 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:45.445250    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:27:46.834744 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:46.834709    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:27:46.894416 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:46.894383    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:27:51.229505 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:51.229449    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-78b5766db6-4r7qh" podUID="74019db4-ab3b-4e74-8cd0-a0fa363732d6" containerName="console" containerID="cri-o://bafed6c06fbae1fd4e6ac12859132fc4299a1f2f4fc0d6368fdc2f243902e0c9" gracePeriod=15
Jul 06 20:27:51.483001 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:51.482934    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78b5766db6-4r7qh"
Jul 06 20:27:51.583834 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:51.583802    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/74019db4-ab3b-4e74-8cd0-a0fa363732d6-oauth-serving-cert\") pod \"74019db4-ab3b-4e74-8cd0-a0fa363732d6\" (UID: \"74019db4-ab3b-4e74-8cd0-a0fa363732d6\") "
Jul 06 20:27:51.584030 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:51.583859    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tzqsq\" (UniqueName: \"kubernetes.io/projected/74019db4-ab3b-4e74-8cd0-a0fa363732d6-kube-api-access-tzqsq\") pod \"74019db4-ab3b-4e74-8cd0-a0fa363732d6\" (UID: \"74019db4-ab3b-4e74-8cd0-a0fa363732d6\") "
Jul 06 20:27:51.584030 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:51.583880    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/74019db4-ab3b-4e74-8cd0-a0fa363732d6-service-ca\") pod \"74019db4-ab3b-4e74-8cd0-a0fa363732d6\" (UID: \"74019db4-ab3b-4e74-8cd0-a0fa363732d6\") "
Jul 06 20:27:51.584030 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:51.583907    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/74019db4-ab3b-4e74-8cd0-a0fa363732d6-console-oauth-config\") pod \"74019db4-ab3b-4e74-8cd0-a0fa363732d6\" (UID: \"74019db4-ab3b-4e74-8cd0-a0fa363732d6\") "
Jul 06 20:27:51.584030 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:51.583926    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/74019db4-ab3b-4e74-8cd0-a0fa363732d6-console-config\") pod \"74019db4-ab3b-4e74-8cd0-a0fa363732d6\" (UID: \"74019db4-ab3b-4e74-8cd0-a0fa363732d6\") "
Jul 06 20:27:51.584030 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:51.583948    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/74019db4-ab3b-4e74-8cd0-a0fa363732d6-console-serving-cert\") pod \"74019db4-ab3b-4e74-8cd0-a0fa363732d6\" (UID: \"74019db4-ab3b-4e74-8cd0-a0fa363732d6\") "
Jul 06 20:27:51.584030 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:51.584007    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74019db4-ab3b-4e74-8cd0-a0fa363732d6-trusted-ca-bundle\") pod \"74019db4-ab3b-4e74-8cd0-a0fa363732d6\" (UID: \"74019db4-ab3b-4e74-8cd0-a0fa363732d6\") "
Jul 06 20:27:51.584355 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:51.584324    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74019db4-ab3b-4e74-8cd0-a0fa363732d6-service-ca" (OuterVolumeSpecName: "service-ca") pod "74019db4-ab3b-4e74-8cd0-a0fa363732d6" (UID: "74019db4-ab3b-4e74-8cd0-a0fa363732d6"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 20:27:51.584411 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:51.584374    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74019db4-ab3b-4e74-8cd0-a0fa363732d6-console-config" (OuterVolumeSpecName: "console-config") pod "74019db4-ab3b-4e74-8cd0-a0fa363732d6" (UID: "74019db4-ab3b-4e74-8cd0-a0fa363732d6"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 20:27:51.584478 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:51.584458    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74019db4-ab3b-4e74-8cd0-a0fa363732d6-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "74019db4-ab3b-4e74-8cd0-a0fa363732d6" (UID: "74019db4-ab3b-4e74-8cd0-a0fa363732d6"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 20:27:51.584529 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:51.584482    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74019db4-ab3b-4e74-8cd0-a0fa363732d6-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "74019db4-ab3b-4e74-8cd0-a0fa363732d6" (UID: "74019db4-ab3b-4e74-8cd0-a0fa363732d6"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 20:27:51.586239 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:51.586214    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74019db4-ab3b-4e74-8cd0-a0fa363732d6-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "74019db4-ab3b-4e74-8cd0-a0fa363732d6" (UID: "74019db4-ab3b-4e74-8cd0-a0fa363732d6"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 20:27:51.586336 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:51.586272    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74019db4-ab3b-4e74-8cd0-a0fa363732d6-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "74019db4-ab3b-4e74-8cd0-a0fa363732d6" (UID: "74019db4-ab3b-4e74-8cd0-a0fa363732d6"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 20:27:51.586336 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:51.586307    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74019db4-ab3b-4e74-8cd0-a0fa363732d6-kube-api-access-tzqsq" (OuterVolumeSpecName: "kube-api-access-tzqsq") pod "74019db4-ab3b-4e74-8cd0-a0fa363732d6" (UID: "74019db4-ab3b-4e74-8cd0-a0fa363732d6"). InnerVolumeSpecName "kube-api-access-tzqsq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 20:27:51.685096 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:51.685048    2563 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/74019db4-ab3b-4e74-8cd0-a0fa363732d6-oauth-serving-cert\") on node \"ip-10-0-141-123.ec2.internal\" DevicePath \"\""
Jul 06 20:27:51.685096 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:51.685091    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzqsq\" (UniqueName: \"kubernetes.io/projected/74019db4-ab3b-4e74-8cd0-a0fa363732d6-kube-api-access-tzqsq\") on node \"ip-10-0-141-123.ec2.internal\" DevicePath \"\""
Jul 06 20:27:51.685096 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:51.685102    2563 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/74019db4-ab3b-4e74-8cd0-a0fa363732d6-service-ca\") on node \"ip-10-0-141-123.ec2.internal\" DevicePath \"\""
Jul 06 20:27:51.685096 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:51.685112    2563 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/74019db4-ab3b-4e74-8cd0-a0fa363732d6-console-oauth-config\") on node \"ip-10-0-141-123.ec2.internal\" DevicePath \"\""
Jul 06 20:27:51.685355 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:51.685121    2563 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/74019db4-ab3b-4e74-8cd0-a0fa363732d6-console-config\") on node \"ip-10-0-141-123.ec2.internal\" DevicePath \"\""
Jul 06 20:27:51.685355 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:51.685129    2563 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/74019db4-ab3b-4e74-8cd0-a0fa363732d6-console-serving-cert\") on node \"ip-10-0-141-123.ec2.internal\" DevicePath \"\""
Jul 06 20:27:51.685355 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:51.685137    2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74019db4-ab3b-4e74-8cd0-a0fa363732d6-trusted-ca-bundle\") on node \"ip-10-0-141-123.ec2.internal\" DevicePath \"\""
Jul 06 20:27:51.999165 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:51.999121    2563 generic.go:356] "Generic (PLEG): container finished" podID="74019db4-ab3b-4e74-8cd0-a0fa363732d6" containerID="bafed6c06fbae1fd4e6ac12859132fc4299a1f2f4fc0d6368fdc2f243902e0c9" exitCode=0
Jul 06 20:27:51.999165 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:51.999165    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78b5766db6-4r7qh"
Jul 06 20:27:51.999379 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:51.999209    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78b5766db6-4r7qh" event={"ID":"74019db4-ab3b-4e74-8cd0-a0fa363732d6","Type":"ContainerDied","Data":"bafed6c06fbae1fd4e6ac12859132fc4299a1f2f4fc0d6368fdc2f243902e0c9"}
Jul 06 20:27:51.999379 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:51.999258    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78b5766db6-4r7qh" event={"ID":"74019db4-ab3b-4e74-8cd0-a0fa363732d6","Type":"ContainerDied","Data":"19a4e56ae4678b2a259b656d93e244dbb0c1bd45d2d3d5d6313730970be317d1"}
Jul 06 20:27:51.999379 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:51.999278    2563 scope.go:117] "RemoveContainer" containerID="bafed6c06fbae1fd4e6ac12859132fc4299a1f2f4fc0d6368fdc2f243902e0c9"
Jul 06 20:27:52.009450 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:52.009427    2563 scope.go:117] "RemoveContainer" containerID="bafed6c06fbae1fd4e6ac12859132fc4299a1f2f4fc0d6368fdc2f243902e0c9"
Jul 06 20:27:52.009743 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:27:52.009703    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bafed6c06fbae1fd4e6ac12859132fc4299a1f2f4fc0d6368fdc2f243902e0c9\": container with ID starting with bafed6c06fbae1fd4e6ac12859132fc4299a1f2f4fc0d6368fdc2f243902e0c9 not found: ID does not exist" containerID="bafed6c06fbae1fd4e6ac12859132fc4299a1f2f4fc0d6368fdc2f243902e0c9"
Jul 06 20:27:52.009829 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:52.009752    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bafed6c06fbae1fd4e6ac12859132fc4299a1f2f4fc0d6368fdc2f243902e0c9"} err="failed to get container status \"bafed6c06fbae1fd4e6ac12859132fc4299a1f2f4fc0d6368fdc2f243902e0c9\": rpc error: code = NotFound desc = could not find container \"bafed6c06fbae1fd4e6ac12859132fc4299a1f2f4fc0d6368fdc2f243902e0c9\": container with ID starting with bafed6c06fbae1fd4e6ac12859132fc4299a1f2f4fc0d6368fdc2f243902e0c9 not found: ID does not exist"
Jul 06 20:27:52.021554 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:52.021530    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-78b5766db6-4r7qh"]
Jul 06 20:27:52.025200 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:52.025182    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-78b5766db6-4r7qh"]
Jul 06 20:27:52.833461 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:27:52.833430    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="74019db4-ab3b-4e74-8cd0-a0fa363732d6" path="/var/lib/kubelet/pods/74019db4-ab3b-4e74-8cd0-a0fa363732d6/volumes"
Jul 06 20:28:11.211688 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:28:11.211656    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:28:11.227340 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:28:11.227309    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:28:12.099088 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:28:12.099063    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:28:19.660425 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:28:19.660337    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:28:19.715305 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:28:19.715270    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:28:31.638439 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:28:31.638403    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:28:31.696424 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:28:31.696395    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:28:35.556384 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:28:35.556348    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:28:35.656870 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:28:35.656838    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:28:35.724899 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:28:35.724868    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:28:42.505104 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:28:42.505066    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:28:42.574019 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:28:42.573985    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:28:42.636780 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:28:42.636746    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:28:42.705662 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:28:42.705622    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:28:43.258288 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:28:43.258255    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:28:43.319783 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:28:43.319753    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:28:53.651118 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:28:53.651078    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:28:53.751806 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:28:53.751771    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:28:53.852988 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:28:53.852955    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:28:56.654012 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:28:56.653986    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-2m7gh_bb2a41aa-a9fc-4ec5-9b78-5e22c1fd301b/console-operator/1.log"
Jul 06 20:28:56.660126 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:28:56.660106    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-2m7gh_bb2a41aa-a9fc-4ec5-9b78-5e22c1fd301b/console-operator/1.log"
Jul 06 20:28:56.668491 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:28:56.668468    2563 kubelet.go:1648] "Image garbage collection succeeded"
Jul 06 20:29:17.016468 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:29:17.016422    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:29:17.183074 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:29:17.183038    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:29:49.428645 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:29:49.428566    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:29:50.880595 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:29:50.880556    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:29:53.421598 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:29:53.421559    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:29:53.523381 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:29:53.523346    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:29:55.041004 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:29:55.040962    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:29:55.151301 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:29:55.151264    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:29:55.377611 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:29:55.377582    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:29:55.574437 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:29:55.574409    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:29:57.341599 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:29:57.341547    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:29:57.522010 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:29:57.521968    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:30:00.185165 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:30:00.185133    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:30:00.289033 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:30:00.288993    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:30:04.937364 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:30:04.937325    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:30:05.497610 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:30:05.497574    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-6z5mv"]
Jul 06 20:30:05.497917 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:30:05.497903    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74019db4-ab3b-4e74-8cd0-a0fa363732d6" containerName="console"
Jul 06 20:30:05.497917 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:30:05.497919    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="74019db4-ab3b-4e74-8cd0-a0fa363732d6" containerName="console"
Jul 06 20:30:05.498046 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:30:05.497969    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="74019db4-ab3b-4e74-8cd0-a0fa363732d6" containerName="console"
Jul 06 20:30:05.500852 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:30:05.500835    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-6z5mv"
Jul 06 20:30:05.503276 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:30:05.503255    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-gcjmq\""
Jul 06 20:30:05.503793 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:30:05.503775    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Jul 06 20:30:05.504020 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:30:05.504003    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Jul 06 20:30:05.510060 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:30:05.510037    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-6z5mv"]
Jul 06 20:30:05.650552 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:30:05.650518    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/face5c55-bfa0-44f6-a6bc-02a8d0a7f574-bound-sa-token\") pod \"cert-manager-55b4d954d5-6z5mv\" (UID: \"face5c55-bfa0-44f6-a6bc-02a8d0a7f574\") " pod="cert-manager/cert-manager-55b4d954d5-6z5mv"
Jul 06 20:30:05.650777 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:30:05.650616    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnjk8\" (UniqueName: \"kubernetes.io/projected/face5c55-bfa0-44f6-a6bc-02a8d0a7f574-kube-api-access-lnjk8\") pod \"cert-manager-55b4d954d5-6z5mv\" (UID: \"face5c55-bfa0-44f6-a6bc-02a8d0a7f574\") " pod="cert-manager/cert-manager-55b4d954d5-6z5mv"
Jul 06 20:30:05.751850 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:30:05.751765    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnjk8\" (UniqueName: \"kubernetes.io/projected/face5c55-bfa0-44f6-a6bc-02a8d0a7f574-kube-api-access-lnjk8\") pod \"cert-manager-55b4d954d5-6z5mv\" (UID: \"face5c55-bfa0-44f6-a6bc-02a8d0a7f574\") " pod="cert-manager/cert-manager-55b4d954d5-6z5mv"
Jul 06 20:30:05.751850 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:30:05.751831    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/face5c55-bfa0-44f6-a6bc-02a8d0a7f574-bound-sa-token\") pod \"cert-manager-55b4d954d5-6z5mv\" (UID: \"face5c55-bfa0-44f6-a6bc-02a8d0a7f574\") " pod="cert-manager/cert-manager-55b4d954d5-6z5mv"
Jul 06 20:30:05.763443 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:30:05.763414    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/face5c55-bfa0-44f6-a6bc-02a8d0a7f574-bound-sa-token\") pod \"cert-manager-55b4d954d5-6z5mv\" (UID: \"face5c55-bfa0-44f6-a6bc-02a8d0a7f574\") " pod="cert-manager/cert-manager-55b4d954d5-6z5mv"
Jul 06 20:30:05.764940 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:30:05.764922    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnjk8\" (UniqueName: \"kubernetes.io/projected/face5c55-bfa0-44f6-a6bc-02a8d0a7f574-kube-api-access-lnjk8\") pod \"cert-manager-55b4d954d5-6z5mv\" (UID: \"face5c55-bfa0-44f6-a6bc-02a8d0a7f574\") " pod="cert-manager/cert-manager-55b4d954d5-6z5mv"
Jul 06 20:30:05.820481 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:30:05.820452    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-6z5mv"
Jul 06 20:30:05.951307 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:30:05.951281    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-6z5mv"]
Jul 06 20:30:05.954354 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:30:05.954325    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podface5c55_bfa0_44f6_a6bc_02a8d0a7f574.slice/crio-c2b685e51897cae5d59f1c50b339578dba25831aeb9703668c6d73111d0ac7c7 WatchSource:0}: Error finding container c2b685e51897cae5d59f1c50b339578dba25831aeb9703668c6d73111d0ac7c7: Status 404 returned error can't find the container with id c2b685e51897cae5d59f1c50b339578dba25831aeb9703668c6d73111d0ac7c7
Jul 06 20:30:05.956303 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:30:05.956287    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 06 20:30:06.524521 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:30:06.524489    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-6z5mv" event={"ID":"face5c55-bfa0-44f6-a6bc-02a8d0a7f574","Type":"ContainerStarted","Data":"c2b685e51897cae5d59f1c50b339578dba25831aeb9703668c6d73111d0ac7c7"}
Jul 06 20:30:07.723179 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:30:07.723149    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:30:07.823745 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:30:07.823692    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:30:08.814965 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:30:08.814928    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:30:08.916617 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:30:08.916571    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:30:13.554934 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:30:13.554889    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-6z5mv" event={"ID":"face5c55-bfa0-44f6-a6bc-02a8d0a7f574","Type":"ContainerStarted","Data":"1f0d9c98c2754f1423c1f92aa31ba70cf773534d1d68fd13420e9874bc9c506a"}
Jul 06 20:30:13.581417 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:30:13.581363    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-6z5mv" podStartSLOduration=1.7646544199999998 podStartE2EDuration="8.581345319s" totalImagesPullingTime="6.816690899s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:30:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:30:13.580460289 +0000 UTC m=+377.296570950" watchObservedRunningTime="2026-07-06 20:30:13.581345319 +0000 UTC m=+377.297455980"
Jul 06 20:30:14.605104 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:30:14.605069    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:30:14.739293 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:30:14.739256    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:30:16.559612 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:30:16.559579    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:30:28.493594 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:30:28.493555    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:30:28.715719 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:30:28.715683    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:30:29.178789 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:30:29.178755    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:30:30.317436 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:30:30.317398    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:30:30.499742 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:30:30.499705    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-656f947554-6f99f"]
Jul 06 20:30:30.503116 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:30:30.503101    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-656f947554-6f99f"
Jul 06 20:30:30.510117 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:30:30.510093    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 06 20:30:30.510244 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:30:30.510123    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 06 20:30:30.510604 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:30:30.510590    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 06 20:30:30.510680 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:30:30.510665    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 06 20:30:30.511322 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:30:30.511308    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-l66h4\""
Jul 06 20:30:30.525376 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:30:30.525346    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:30:30.547305 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:30:30.547282    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-656f947554-6f99f"]
Jul 06 20:30:30.553977 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:30:30.553956    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f29b869f-09ec-4619-8eb3-ab4fb32c7afb-tmp\") pod \"opendatahub-operator-controller-manager-656f947554-6f99f\" (UID: \"f29b869f-09ec-4619-8eb3-ab4fb32c7afb\") " pod="opendatahub/opendatahub-operator-controller-manager-656f947554-6f99f"
Jul 06 20:30:30.554080 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:30:30.553988    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f29b869f-09ec-4619-8eb3-ab4fb32c7afb-webhook-cert\") pod \"opendatahub-operator-controller-manager-656f947554-6f99f\" (UID: \"f29b869f-09ec-4619-8eb3-ab4fb32c7afb\") " pod="opendatahub/opendatahub-operator-controller-manager-656f947554-6f99f"
Jul 06 20:30:30.554080 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:30:30.554019    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f29b869f-09ec-4619-8eb3-ab4fb32c7afb-manifests\") pod \"opendatahub-operator-controller-manager-656f947554-6f99f\" (UID: \"f29b869f-09ec-4619-8eb3-ab4fb32c7afb\") " pod="opendatahub/opendatahub-operator-controller-manager-656f947554-6f99f"
Jul 06 20:30:30.554080 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:30:30.554036    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f29b869f-09ec-4619-8eb3-ab4fb32c7afb-apiservice-cert\") pod \"opendatahub-operator-controller-manager-656f947554-6f99f\" (UID: \"f29b869f-09ec-4619-8eb3-ab4fb32c7afb\") " pod="opendatahub/opendatahub-operator-controller-manager-656f947554-6f99f"
Jul 06 20:30:30.554179 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:30:30.554145    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-db79f\" (UniqueName: \"kubernetes.io/projected/f29b869f-09ec-4619-8eb3-ab4fb32c7afb-kube-api-access-db79f\") pod \"opendatahub-operator-controller-manager-656f947554-6f99f\" (UID: \"f29b869f-09ec-4619-8eb3-ab4fb32c7afb\") " pod="opendatahub/opendatahub-operator-controller-manager-656f947554-6f99f"
Jul 06 20:30:30.655232 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:30:30.655149    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-db79f\" (UniqueName: \"kubernetes.io/projected/f29b869f-09ec-4619-8eb3-ab4fb32c7afb-kube-api-access-db79f\") pod \"opendatahub-operator-controller-manager-656f947554-6f99f\" (UID: \"f29b869f-09ec-4619-8eb3-ab4fb32c7afb\") " pod="opendatahub/opendatahub-operator-controller-manager-656f947554-6f99f"
Jul 06 20:30:30.655232 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:30:30.655208    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f29b869f-09ec-4619-8eb3-ab4fb32c7afb-tmp\") pod \"opendatahub-operator-controller-manager-656f947554-6f99f\" (UID: \"f29b869f-09ec-4619-8eb3-ab4fb32c7afb\") " pod="opendatahub/opendatahub-operator-controller-manager-656f947554-6f99f"
Jul 06 20:30:30.655232 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:30:30.655235    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f29b869f-09ec-4619-8eb3-ab4fb32c7afb-webhook-cert\") pod \"opendatahub-operator-controller-manager-656f947554-6f99f\" (UID: \"f29b869f-09ec-4619-8eb3-ab4fb32c7afb\") " pod="opendatahub/opendatahub-operator-controller-manager-656f947554-6f99f"
Jul 06 20:30:30.655470 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:30:30.655260    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f29b869f-09ec-4619-8eb3-ab4fb32c7afb-manifests\") pod \"opendatahub-operator-controller-manager-656f947554-6f99f\" (UID: \"f29b869f-09ec-4619-8eb3-ab4fb32c7afb\") " pod="opendatahub/opendatahub-operator-controller-manager-656f947554-6f99f"
Jul 06 20:30:30.655470 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:30:30.655279    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f29b869f-09ec-4619-8eb3-ab4fb32c7afb-apiservice-cert\") pod \"opendatahub-operator-controller-manager-656f947554-6f99f\" (UID: \"f29b869f-09ec-4619-8eb3-ab4fb32c7afb\") " pod="opendatahub/opendatahub-operator-controller-manager-656f947554-6f99f"
Jul 06 20:30:30.655666 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:30:30.655642    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f29b869f-09ec-4619-8eb3-ab4fb32c7afb-manifests\") pod \"opendatahub-operator-controller-manager-656f947554-6f99f\" (UID: \"f29b869f-09ec-4619-8eb3-ab4fb32c7afb\") " pod="opendatahub/opendatahub-operator-controller-manager-656f947554-6f99f"
Jul 06 20:30:30.657815 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:30:30.657788    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f29b869f-09ec-4619-8eb3-ab4fb32c7afb-apiservice-cert\") pod \"opendatahub-operator-controller-manager-656f947554-6f99f\" (UID: \"f29b869f-09ec-4619-8eb3-ab4fb32c7afb\") " pod="opendatahub/opendatahub-operator-controller-manager-656f947554-6f99f"
Jul 06 20:30:30.657815 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:30:30.657811    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f29b869f-09ec-4619-8eb3-ab4fb32c7afb-tmp\") pod \"opendatahub-operator-controller-manager-656f947554-6f99f\" (UID: \"f29b869f-09ec-4619-8eb3-ab4fb32c7afb\") " pod="opendatahub/opendatahub-operator-controller-manager-656f947554-6f99f"
Jul 06 20:30:30.658023 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:30:30.658006    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f29b869f-09ec-4619-8eb3-ab4fb32c7afb-webhook-cert\") pod \"opendatahub-operator-controller-manager-656f947554-6f99f\" (UID: \"f29b869f-09ec-4619-8eb3-ab4fb32c7afb\") " pod="opendatahub/opendatahub-operator-controller-manager-656f947554-6f99f"
Jul 06 20:30:30.682499 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:30:30.682475    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-db79f\" (UniqueName: \"kubernetes.io/projected/f29b869f-09ec-4619-8eb3-ab4fb32c7afb-kube-api-access-db79f\") pod \"opendatahub-operator-controller-manager-656f947554-6f99f\" (UID: \"f29b869f-09ec-4619-8eb3-ab4fb32c7afb\") " pod="opendatahub/opendatahub-operator-controller-manager-656f947554-6f99f"
Jul 06 20:30:30.813390 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:30:30.813350    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-656f947554-6f99f"
Jul 06 20:30:30.982647 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:30:30.982567    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-656f947554-6f99f"]
Jul 06 20:30:30.986217 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:30:30.986184    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf29b869f_09ec_4619_8eb3_ab4fb32c7afb.slice/crio-2859cc49d3704ddcba68d2599a1f38414d033b8a68fc957e9da64ed5e493c29f WatchSource:0}: Error finding container 2859cc49d3704ddcba68d2599a1f38414d033b8a68fc957e9da64ed5e493c29f: Status 404 returned error can't find the container with id 2859cc49d3704ddcba68d2599a1f38414d033b8a68fc957e9da64ed5e493c29f
Jul 06 20:30:31.035249 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:30:31.035219    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:30:31.196413 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:30:31.196375    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:30:31.425597 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:30:31.425564    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:30:31.577318 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:30:31.577284    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:30:31.627743 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:30:31.627692    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-656f947554-6f99f" event={"ID":"f29b869f-09ec-4619-8eb3-ab4fb32c7afb","Type":"ContainerStarted","Data":"2859cc49d3704ddcba68d2599a1f38414d033b8a68fc957e9da64ed5e493c29f"}
Jul 06 20:30:31.756265 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:30:31.756173    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:30:34.646797 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:30:34.646762    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-656f947554-6f99f" event={"ID":"f29b869f-09ec-4619-8eb3-ab4fb32c7afb","Type":"ContainerStarted","Data":"507532b9f674d2fefbf06042fcc364fc9dade591671c795c77e3c27ec97a5bb0"}
Jul 06 20:30:35.652514 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:30:35.652481    2563 generic.go:356] "Generic (PLEG): container finished" podID="f29b869f-09ec-4619-8eb3-ab4fb32c7afb" containerID="507532b9f674d2fefbf06042fcc364fc9dade591671c795c77e3c27ec97a5bb0" exitCode=0
Jul 06 20:30:35.652935 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:30:35.652522    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-656f947554-6f99f" event={"ID":"f29b869f-09ec-4619-8eb3-ab4fb32c7afb","Type":"ContainerDied","Data":"507532b9f674d2fefbf06042fcc364fc9dade591671c795c77e3c27ec97a5bb0"}
Jul 06 20:30:36.658351 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:30:36.658315    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-656f947554-6f99f" event={"ID":"f29b869f-09ec-4619-8eb3-ab4fb32c7afb","Type":"ContainerStarted","Data":"b29295bfad8c6a72ea540dbc44bdc09c4ed52884b6cbfc3d6ac09d9e2327407c"}
Jul 06 20:30:36.658805 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:30:36.658421    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-656f947554-6f99f"
Jul 06 20:30:36.724342 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:30:36.724282    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-656f947554-6f99f" podStartSLOduration=2.34225598 podStartE2EDuration="6.724263045s" totalImagesPullingTime="3.295750955s" totalInitContainerRuntime="1.08625611s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:30:30 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:30:36.722355444 +0000 UTC m=+400.438466104" watchObservedRunningTime="2026-07-06 20:30:36.724263045 +0000 UTC m=+400.440373707"
Jul 06 20:30:37.356263 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:30:37.356230    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:30:37.468062 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:30:37.467977    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:30:44.460087 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:30:44.460050    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:30:44.899026 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:30:44.898971    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:30:44.954031 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:30:44.953998    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:30:47.664423 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:30:47.664398    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-656f947554-6f99f"
Jul 06 20:30:49.881994 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:30:49.881953    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:30:50.074308 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:30:50.074264    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:30:50.620493 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:30:50.620458    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:30:50.927146 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:30:50.927020    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:30:50.927605 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:30:50.927145    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-65d4f4589-nnmqp"]
Jul 06 20:30:50.930130 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:30:50.930102    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-65d4f4589-nnmqp"
Jul 06 20:30:50.934369 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:30:50.934346    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Jul 06 20:30:50.935089 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:30:50.935062    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 06 20:30:50.935089 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:30:50.935082    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 06 20:30:50.935290 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:30:50.935208    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 06 20:30:50.935455 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:30:50.935436    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-qrt85\""
Jul 06 20:30:50.966861 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:30:50.966831    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-65d4f4589-nnmqp"]
Jul 06 20:30:51.038518 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:30:51.038486    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svjfg\" (UniqueName: \"kubernetes.io/projected/5a6b0ea0-6a10-4a01-97c8-11d180c63547-kube-api-access-svjfg\") pod \"kube-auth-proxy-65d4f4589-nnmqp\" (UID: \"5a6b0ea0-6a10-4a01-97c8-11d180c63547\") " pod="openshift-ingress/kube-auth-proxy-65d4f4589-nnmqp"
Jul 06 20:30:51.038518 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:30:51.038530    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a6b0ea0-6a10-4a01-97c8-11d180c63547-tls-certs\") pod \"kube-auth-proxy-65d4f4589-nnmqp\" (UID: \"5a6b0ea0-6a10-4a01-97c8-11d180c63547\") " pod="openshift-ingress/kube-auth-proxy-65d4f4589-nnmqp"
Jul 06 20:30:51.038764 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:30:51.038546    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5a6b0ea0-6a10-4a01-97c8-11d180c63547-tmp\") pod \"kube-auth-proxy-65d4f4589-nnmqp\" (UID: \"5a6b0ea0-6a10-4a01-97c8-11d180c63547\") " pod="openshift-ingress/kube-auth-proxy-65d4f4589-nnmqp"
Jul 06 20:30:51.139999 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:30:51.139961    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svjfg\" (UniqueName: \"kubernetes.io/projected/5a6b0ea0-6a10-4a01-97c8-11d180c63547-kube-api-access-svjfg\") pod \"kube-auth-proxy-65d4f4589-nnmqp\" (UID: \"5a6b0ea0-6a10-4a01-97c8-11d180c63547\") " pod="openshift-ingress/kube-auth-proxy-65d4f4589-nnmqp"
Jul 06 20:30:51.140213 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:30:51.140011    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a6b0ea0-6a10-4a01-97c8-11d180c63547-tls-certs\") pod \"kube-auth-proxy-65d4f4589-nnmqp\" (UID: \"5a6b0ea0-6a10-4a01-97c8-11d180c63547\") " pod="openshift-ingress/kube-auth-proxy-65d4f4589-nnmqp"
Jul 06 20:30:51.140213 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:30:51.140035    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5a6b0ea0-6a10-4a01-97c8-11d180c63547-tmp\") pod \"kube-auth-proxy-65d4f4589-nnmqp\" (UID: \"5a6b0ea0-6a10-4a01-97c8-11d180c63547\") " pod="openshift-ingress/kube-auth-proxy-65d4f4589-nnmqp"
Jul 06 20:30:51.142384 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:30:51.142361    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5a6b0ea0-6a10-4a01-97c8-11d180c63547-tmp\") pod \"kube-auth-proxy-65d4f4589-nnmqp\" (UID: \"5a6b0ea0-6a10-4a01-97c8-11d180c63547\") " pod="openshift-ingress/kube-auth-proxy-65d4f4589-nnmqp"
Jul 06 20:30:51.142702 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:30:51.142680    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a6b0ea0-6a10-4a01-97c8-11d180c63547-tls-certs\") pod \"kube-auth-proxy-65d4f4589-nnmqp\" (UID: \"5a6b0ea0-6a10-4a01-97c8-11d180c63547\") " pod="openshift-ingress/kube-auth-proxy-65d4f4589-nnmqp"
Jul 06 20:30:51.159262 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:30:51.159231    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svjfg\" (UniqueName: \"kubernetes.io/projected/5a6b0ea0-6a10-4a01-97c8-11d180c63547-kube-api-access-svjfg\") pod \"kube-auth-proxy-65d4f4589-nnmqp\" (UID: \"5a6b0ea0-6a10-4a01-97c8-11d180c63547\") " pod="openshift-ingress/kube-auth-proxy-65d4f4589-nnmqp"
Jul 06 20:30:51.206501 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:30:51.206393    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:30:51.242508 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:30:51.242480    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-65d4f4589-nnmqp"
Jul 06 20:30:51.374175 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:30:51.374144    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:30:51.425123 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:30:51.425011    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-65d4f4589-nnmqp"]
Jul 06 20:30:51.427918 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:30:51.427891    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a6b0ea0_6a10_4a01_97c8_11d180c63547.slice/crio-fee6dca6cf75e51669875d4208fca0c8a615bac356e7a4c37f18f9f67f2b787c WatchSource:0}: Error finding container fee6dca6cf75e51669875d4208fca0c8a615bac356e7a4c37f18f9f67f2b787c: Status 404 returned error can't find the container with id fee6dca6cf75e51669875d4208fca0c8a615bac356e7a4c37f18f9f67f2b787c
Jul 06 20:30:51.499964 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:30:51.499817    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:30:51.725617 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:30:51.725586    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-65d4f4589-nnmqp" event={"ID":"5a6b0ea0-6a10-4a01-97c8-11d180c63547","Type":"ContainerStarted","Data":"fee6dca6cf75e51669875d4208fca0c8a615bac356e7a4c37f18f9f67f2b787c"}
Jul 06 20:30:52.004383 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:30:52.004339    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:30:52.532217 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:30:52.532159    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:30:52.711878 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:30:52.711835    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:30:52.818087 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:30:52.818053    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:30:54.324960 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:30:54.324922    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:30:55.746746 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:30:55.746689    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-65d4f4589-nnmqp" event={"ID":"5a6b0ea0-6a10-4a01-97c8-11d180c63547","Type":"ContainerStarted","Data":"dfc297975a58847f539c542859e78387c42ee52082973aac0b8316b075a88a4b"}
Jul 06 20:30:55.766896 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:30:55.766850    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-65d4f4589-nnmqp" podStartSLOduration=2.411836775 podStartE2EDuration="5.766834928s" totalImagesPullingTime="3.354998153s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:30:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:30:55.76581908 +0000 UTC m=+419.481929743" watchObservedRunningTime="2026-07-06 20:30:55.766834928 +0000 UTC m=+419.482945589"
Jul 06 20:30:56.066208 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:30:56.066178    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:30:56.170500 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:30:56.170445    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:30:57.035949 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:30:57.035916    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:30:57.335237 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:30:57.335202    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:30:57.408791 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:30:57.408752    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:31:00.958506 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:31:00.958462    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-6cf865c78f-phbhx"]
Jul 06 20:31:00.963529 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:31:00.963505    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6cf865c78f-phbhx"
Jul 06 20:31:00.971323 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:31:00.971303    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 06 20:31:00.973001 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:31:00.972985    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-wbnmh\""
Jul 06 20:31:00.974392 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:31:00.974379    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 06 20:31:00.985319 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:31:00.985303    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 06 20:31:01.001496 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:31:01.001467    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 06 20:31:01.001788 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:31:01.001773    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 06 20:31:01.003645 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:31:01.003620    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6cf865c78f-phbhx"]
Jul 06 20:31:01.130110 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:31:01.130080    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2e486719-5dc2-48d2-ac78-293db8913c7c-cert\") pod \"lws-controller-manager-6cf865c78f-phbhx\" (UID: \"2e486719-5dc2-48d2-ac78-293db8913c7c\") " pod="openshift-lws-operator/lws-controller-manager-6cf865c78f-phbhx"
Jul 06 20:31:01.130308 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:31:01.130118    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4nkc\" (UniqueName: \"kubernetes.io/projected/2e486719-5dc2-48d2-ac78-293db8913c7c-kube-api-access-h4nkc\") pod \"lws-controller-manager-6cf865c78f-phbhx\" (UID: \"2e486719-5dc2-48d2-ac78-293db8913c7c\") " pod="openshift-lws-operator/lws-controller-manager-6cf865c78f-phbhx"
Jul 06 20:31:01.130308 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:31:01.130148    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/2e486719-5dc2-48d2-ac78-293db8913c7c-metrics-cert\") pod \"lws-controller-manager-6cf865c78f-phbhx\" (UID: \"2e486719-5dc2-48d2-ac78-293db8913c7c\") " pod="openshift-lws-operator/lws-controller-manager-6cf865c78f-phbhx"
Jul 06 20:31:01.130308 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:31:01.130245    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/2e486719-5dc2-48d2-ac78-293db8913c7c-manager-config\") pod \"lws-controller-manager-6cf865c78f-phbhx\" (UID: \"2e486719-5dc2-48d2-ac78-293db8913c7c\") " pod="openshift-lws-operator/lws-controller-manager-6cf865c78f-phbhx"
Jul 06 20:31:01.231159 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:31:01.231081    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2e486719-5dc2-48d2-ac78-293db8913c7c-cert\") pod \"lws-controller-manager-6cf865c78f-phbhx\" (UID: \"2e486719-5dc2-48d2-ac78-293db8913c7c\") " pod="openshift-lws-operator/lws-controller-manager-6cf865c78f-phbhx"
Jul 06 20:31:01.231159 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:31:01.231124    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4nkc\" (UniqueName: \"kubernetes.io/projected/2e486719-5dc2-48d2-ac78-293db8913c7c-kube-api-access-h4nkc\") pod \"lws-controller-manager-6cf865c78f-phbhx\" (UID: \"2e486719-5dc2-48d2-ac78-293db8913c7c\") " pod="openshift-lws-operator/lws-controller-manager-6cf865c78f-phbhx"
Jul 06 20:31:01.231378 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:31:01.231291    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/2e486719-5dc2-48d2-ac78-293db8913c7c-metrics-cert\") pod \"lws-controller-manager-6cf865c78f-phbhx\" (UID: \"2e486719-5dc2-48d2-ac78-293db8913c7c\") " pod="openshift-lws-operator/lws-controller-manager-6cf865c78f-phbhx"
Jul 06 20:31:01.231378 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:31:01.231355    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/2e486719-5dc2-48d2-ac78-293db8913c7c-manager-config\") pod \"lws-controller-manager-6cf865c78f-phbhx\" (UID: \"2e486719-5dc2-48d2-ac78-293db8913c7c\") " pod="openshift-lws-operator/lws-controller-manager-6cf865c78f-phbhx"
Jul 06 20:31:01.231962 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:31:01.231940    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/2e486719-5dc2-48d2-ac78-293db8913c7c-manager-config\") pod \"lws-controller-manager-6cf865c78f-phbhx\" (UID: \"2e486719-5dc2-48d2-ac78-293db8913c7c\") " pod="openshift-lws-operator/lws-controller-manager-6cf865c78f-phbhx"
Jul 06 20:31:01.233373 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:31:01.233355    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2e486719-5dc2-48d2-ac78-293db8913c7c-cert\") pod \"lws-controller-manager-6cf865c78f-phbhx\" (UID: \"2e486719-5dc2-48d2-ac78-293db8913c7c\") " pod="openshift-lws-operator/lws-controller-manager-6cf865c78f-phbhx"
Jul 06 20:31:01.233614 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:31:01.233592    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/2e486719-5dc2-48d2-ac78-293db8913c7c-metrics-cert\") pod \"lws-controller-manager-6cf865c78f-phbhx\" (UID: \"2e486719-5dc2-48d2-ac78-293db8913c7c\") " pod="openshift-lws-operator/lws-controller-manager-6cf865c78f-phbhx"
Jul 06 20:31:01.244078 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:31:01.244057    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4nkc\" (UniqueName: \"kubernetes.io/projected/2e486719-5dc2-48d2-ac78-293db8913c7c-kube-api-access-h4nkc\") pod \"lws-controller-manager-6cf865c78f-phbhx\" (UID: \"2e486719-5dc2-48d2-ac78-293db8913c7c\") " pod="openshift-lws-operator/lws-controller-manager-6cf865c78f-phbhx"
Jul 06 20:31:01.273051 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:31:01.273015    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6cf865c78f-phbhx"
Jul 06 20:31:01.431248 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:31:01.431208    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6cf865c78f-phbhx"]
Jul 06 20:31:01.434898 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:31:01.434839    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e486719_5dc2_48d2_ac78_293db8913c7c.slice/crio-b6a45d5a5871949f1fa29903398d7c2807e98ed8d88d6d5b759b62530a3832b5 WatchSource:0}: Error finding container b6a45d5a5871949f1fa29903398d7c2807e98ed8d88d6d5b759b62530a3832b5: Status 404 returned error can't find the container with id b6a45d5a5871949f1fa29903398d7c2807e98ed8d88d6d5b759b62530a3832b5
Jul 06 20:31:01.774660 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:31:01.774623    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6cf865c78f-phbhx" event={"ID":"2e486719-5dc2-48d2-ac78-293db8913c7c","Type":"ContainerStarted","Data":"b6a45d5a5871949f1fa29903398d7c2807e98ed8d88d6d5b759b62530a3832b5"}
Jul 06 20:31:02.661605 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:31:02.661533    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:31:02.960075 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:31:02.959900    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:31:03.105360 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:31:03.105329    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:31:03.306876 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:31:03.306845    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:31:03.564291 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:31:03.564204    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:31:03.729176 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:31:03.729141    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:31:03.868696 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:31:03.868600    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:31:04.792110 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:31:04.792018    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6cf865c78f-phbhx" event={"ID":"2e486719-5dc2-48d2-ac78-293db8913c7c","Type":"ContainerStarted","Data":"52cf5165355c2075dd792050d7ebab656fd4dd86b4abb7cd666c3002209e78c3"}
Jul 06 20:31:04.792837 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:31:04.792813    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-6cf865c78f-phbhx"
Jul 06 20:31:04.821655 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:31:04.821594    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-6cf865c78f-phbhx" podStartSLOduration=1.9234958500000001 podStartE2EDuration="4.821572845s" totalImagesPullingTime="2.898076995s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:31:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:31:04.820304739 +0000 UTC m=+428.536415402" watchObservedRunningTime="2026-07-06 20:31:04.821572845 +0000 UTC m=+428.537683508"
Jul 06 20:31:16.801527 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:31:16.801491    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-6cf865c78f-phbhx"
Jul 06 20:31:51.358029 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:31:51.357997    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:31:51.510717 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:31:51.510684    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:31:51.617955 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:31:51.617862    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:32:02.918964 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:32:02.918913    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:32:19.555267 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:32:19.555228    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:32:19.655232 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:32:19.655202    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:32:20.268450 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:32:20.268413    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-65f6bc98bb-d9fxk"]
Jul 06 20:32:20.271962 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:32:20.271937    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65f6bc98bb-d9fxk"
Jul 06 20:32:20.275514 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:32:20.275493    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 06 20:32:20.275678 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:32:20.275658    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 06 20:32:20.275771 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:32:20.275662    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-g2hph\""
Jul 06 20:32:20.275771 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:32:20.275767    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 06 20:32:20.276019 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:32:20.275999    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 06 20:32:20.276125 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:32:20.276067    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 06 20:32:20.276125 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:32:20.276084    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 06 20:32:20.276466 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:32:20.276446    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 06 20:32:20.281564 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:32:20.281548    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 06 20:32:20.292027 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:32:20.292008    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-65f6bc98bb-d9fxk"]
Jul 06 20:32:20.345055 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:32:20.345017    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:32:20.437722 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:32:20.437688    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbq57\" (UniqueName: \"kubernetes.io/projected/66f2e86a-a524-469e-98d2-49ea841709f4-kube-api-access-wbq57\") pod \"console-65f6bc98bb-d9fxk\" (UID: \"66f2e86a-a524-469e-98d2-49ea841709f4\") " pod="openshift-console/console-65f6bc98bb-d9fxk"
Jul 06 20:32:20.437926 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:32:20.437791    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/66f2e86a-a524-469e-98d2-49ea841709f4-console-serving-cert\") pod \"console-65f6bc98bb-d9fxk\" (UID: \"66f2e86a-a524-469e-98d2-49ea841709f4\") " pod="openshift-console/console-65f6bc98bb-d9fxk"
Jul 06 20:32:20.437926 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:32:20.437856    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/66f2e86a-a524-469e-98d2-49ea841709f4-console-oauth-config\") pod \"console-65f6bc98bb-d9fxk\" (UID: \"66f2e86a-a524-469e-98d2-49ea841709f4\") " pod="openshift-console/console-65f6bc98bb-d9fxk"
Jul 06 20:32:20.437926 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:32:20.437874    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66f2e86a-a524-469e-98d2-49ea841709f4-trusted-ca-bundle\") pod \"console-65f6bc98bb-d9fxk\" (UID: \"66f2e86a-a524-469e-98d2-49ea841709f4\") " pod="openshift-console/console-65f6bc98bb-d9fxk"
Jul 06 20:32:20.437926 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:32:20.437902    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/66f2e86a-a524-469e-98d2-49ea841709f4-service-ca\") pod \"console-65f6bc98bb-d9fxk\" (UID: \"66f2e86a-a524-469e-98d2-49ea841709f4\") " pod="openshift-console/console-65f6bc98bb-d9fxk"
Jul 06 20:32:20.437926 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:32:20.437923    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/66f2e86a-a524-469e-98d2-49ea841709f4-oauth-serving-cert\") pod \"console-65f6bc98bb-d9fxk\" (UID: \"66f2e86a-a524-469e-98d2-49ea841709f4\") " pod="openshift-console/console-65f6bc98bb-d9fxk"
Jul 06 20:32:20.438112 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:32:20.437994    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/66f2e86a-a524-469e-98d2-49ea841709f4-console-config\") pod \"console-65f6bc98bb-d9fxk\" (UID: \"66f2e86a-a524-469e-98d2-49ea841709f4\") " pod="openshift-console/console-65f6bc98bb-d9fxk"
Jul 06 20:32:20.450641 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:32:20.450614    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:32:20.539382 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:32:20.539293    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbq57\" (UniqueName: \"kubernetes.io/projected/66f2e86a-a524-469e-98d2-49ea841709f4-kube-api-access-wbq57\") pod \"console-65f6bc98bb-d9fxk\" (UID: \"66f2e86a-a524-469e-98d2-49ea841709f4\") " pod="openshift-console/console-65f6bc98bb-d9fxk"
Jul 06 20:32:20.539382 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:32:20.539369    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/66f2e86a-a524-469e-98d2-49ea841709f4-console-serving-cert\") pod \"console-65f6bc98bb-d9fxk\" (UID: \"66f2e86a-a524-469e-98d2-49ea841709f4\") " pod="openshift-console/console-65f6bc98bb-d9fxk"
Jul 06 20:32:20.539629 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:32:20.539401    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/66f2e86a-a524-469e-98d2-49ea841709f4-console-oauth-config\") pod \"console-65f6bc98bb-d9fxk\" (UID: \"66f2e86a-a524-469e-98d2-49ea841709f4\") " pod="openshift-console/console-65f6bc98bb-d9fxk"
Jul 06 20:32:20.539629 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:32:20.539423    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66f2e86a-a524-469e-98d2-49ea841709f4-trusted-ca-bundle\") pod \"console-65f6bc98bb-d9fxk\" (UID: \"66f2e86a-a524-469e-98d2-49ea841709f4\") " pod="openshift-console/console-65f6bc98bb-d9fxk"
Jul 06 20:32:20.539629 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:32:20.539464    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/66f2e86a-a524-469e-98d2-49ea841709f4-service-ca\") pod \"console-65f6bc98bb-d9fxk\" (UID: \"66f2e86a-a524-469e-98d2-49ea841709f4\") " pod="openshift-console/console-65f6bc98bb-d9fxk"
Jul 06 20:32:20.539629 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:32:20.539492    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/66f2e86a-a524-469e-98d2-49ea841709f4-oauth-serving-cert\") pod \"console-65f6bc98bb-d9fxk\" (UID: \"66f2e86a-a524-469e-98d2-49ea841709f4\") " pod="openshift-console/console-65f6bc98bb-d9fxk"
Jul 06 20:32:20.539629 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:32:20.539556    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/66f2e86a-a524-469e-98d2-49ea841709f4-console-config\") pod \"console-65f6bc98bb-d9fxk\" (UID: \"66f2e86a-a524-469e-98d2-49ea841709f4\") " pod="openshift-console/console-65f6bc98bb-d9fxk"
Jul 06 20:32:20.540339 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:32:20.540313    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/66f2e86a-a524-469e-98d2-49ea841709f4-service-ca\") pod \"console-65f6bc98bb-d9fxk\" (UID: \"66f2e86a-a524-469e-98d2-49ea841709f4\") " pod="openshift-console/console-65f6bc98bb-d9fxk"
Jul 06 20:32:20.540458 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:32:20.540319    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/66f2e86a-a524-469e-98d2-49ea841709f4-oauth-serving-cert\") pod \"console-65f6bc98bb-d9fxk\" (UID: \"66f2e86a-a524-469e-98d2-49ea841709f4\") " pod="openshift-console/console-65f6bc98bb-d9fxk"
Jul 06 20:32:20.540458 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:32:20.540435    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/66f2e86a-a524-469e-98d2-49ea841709f4-console-config\") pod \"console-65f6bc98bb-d9fxk\" (UID: \"66f2e86a-a524-469e-98d2-49ea841709f4\") " pod="openshift-console/console-65f6bc98bb-d9fxk"
Jul 06 20:32:20.540536 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:32:20.540497    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66f2e86a-a524-469e-98d2-49ea841709f4-trusted-ca-bundle\") pod \"console-65f6bc98bb-d9fxk\" (UID: \"66f2e86a-a524-469e-98d2-49ea841709f4\") " pod="openshift-console/console-65f6bc98bb-d9fxk"
Jul 06 20:32:20.542045 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:32:20.542027    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/66f2e86a-a524-469e-98d2-49ea841709f4-console-serving-cert\") pod \"console-65f6bc98bb-d9fxk\" (UID: \"66f2e86a-a524-469e-98d2-49ea841709f4\") " pod="openshift-console/console-65f6bc98bb-d9fxk"
Jul 06 20:32:20.542188 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:32:20.542033    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/66f2e86a-a524-469e-98d2-49ea841709f4-console-oauth-config\") pod \"console-65f6bc98bb-d9fxk\" (UID: \"66f2e86a-a524-469e-98d2-49ea841709f4\") " pod="openshift-console/console-65f6bc98bb-d9fxk"
Jul 06 20:32:20.545769 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:32:20.545724    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:32:20.548223 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:32:20.548207    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbq57\" (UniqueName: \"kubernetes.io/projected/66f2e86a-a524-469e-98d2-49ea841709f4-kube-api-access-wbq57\") pod \"console-65f6bc98bb-d9fxk\" (UID: \"66f2e86a-a524-469e-98d2-49ea841709f4\") " pod="openshift-console/console-65f6bc98bb-d9fxk"
Jul 06 20:32:20.581934 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:32:20.581913    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65f6bc98bb-d9fxk"
Jul 06 20:32:20.649643 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:32:20.649581    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:32:20.717781 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:32:20.717724    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-65f6bc98bb-d9fxk"]
Jul 06 20:32:20.719926 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:32:20.719901    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66f2e86a_a524_469e_98d2_49ea841709f4.slice/crio-3df5b8adbcb43b0dd86edfe886dc07278d36566504adc37770fe25c1b98c729f WatchSource:0}: Error finding container 3df5b8adbcb43b0dd86edfe886dc07278d36566504adc37770fe25c1b98c729f: Status 404 returned error can't find the container with id 3df5b8adbcb43b0dd86edfe886dc07278d36566504adc37770fe25c1b98c729f
Jul 06 20:32:21.069622 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:32:21.069591    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:32:21.121987 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:32:21.121948    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65f6bc98bb-d9fxk" event={"ID":"66f2e86a-a524-469e-98d2-49ea841709f4","Type":"ContainerStarted","Data":"5cd1e39e888add28f67ef52573dd754902a36570d9a089f151b085d6e5aa427b"}
Jul 06 20:32:21.122155 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:32:21.122005    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65f6bc98bb-d9fxk" event={"ID":"66f2e86a-a524-469e-98d2-49ea841709f4","Type":"ContainerStarted","Data":"3df5b8adbcb43b0dd86edfe886dc07278d36566504adc37770fe25c1b98c729f"}
Jul 06 20:32:21.142838 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:32:21.142788    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-65f6bc98bb-d9fxk" podStartSLOduration=1.1427709369999999 podStartE2EDuration="1.142770937s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:32:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:32:21.141931325 +0000 UTC m=+504.858041990" watchObservedRunningTime="2026-07-06 20:32:21.142770937 +0000 UTC m=+504.858881599"
Jul 06 20:32:21.152981 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:32:21.152945    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:32:21.514813 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:32:21.514709    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:32:25.088927 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:32:25.088886    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:32:28.192211 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:32:28.192182    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:32:29.698761 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:32:29.698711    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:32:29.900022 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:32:29.899983    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:32:30.490854 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:32:30.490808    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:32:30.582613 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:32:30.582583    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-65f6bc98bb-d9fxk"
Jul 06 20:32:30.582841 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:32:30.582660    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-65f6bc98bb-d9fxk"
Jul 06 20:32:30.587606 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:32:30.587586    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-65f6bc98bb-d9fxk"
Jul 06 20:32:31.172315 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:32:31.172289    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-65f6bc98bb-d9fxk"
Jul 06 20:32:31.397181 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:32:31.397148    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:32:31.500771 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:32:31.500672    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:32:33.012345 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:32:33.012303    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:32:33.024530 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:32:33.024499    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-lf8jx"]
Jul 06 20:32:33.028307 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:32:33.028287    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-lf8jx"
Jul 06 20:32:33.031541 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:32:33.031522    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-rbrd7\""
Jul 06 20:32:33.032087 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:32:33.032074    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 06 20:32:33.032303 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:32:33.032290    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 06 20:32:33.054284 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:32:33.054257    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-lf8jx"]
Jul 06 20:32:33.156714 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:32:33.156678    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zst7k\" (UniqueName: \"kubernetes.io/projected/70d76a1d-0156-4342-9631-c5ce326197bb-kube-api-access-zst7k\") pod \"authorino-operator-75cc8d5cb7-lf8jx\" (UID: \"70d76a1d-0156-4342-9631-c5ce326197bb\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-lf8jx"
Jul 06 20:32:33.182978 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:32:33.182949    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:32:33.257689 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:32:33.257657    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zst7k\" (UniqueName: \"kubernetes.io/projected/70d76a1d-0156-4342-9631-c5ce326197bb-kube-api-access-zst7k\") pod \"authorino-operator-75cc8d5cb7-lf8jx\" (UID: \"70d76a1d-0156-4342-9631-c5ce326197bb\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-lf8jx"
Jul 06 20:32:33.267664 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:32:33.267611    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zst7k\" (UniqueName: \"kubernetes.io/projected/70d76a1d-0156-4342-9631-c5ce326197bb-kube-api-access-zst7k\") pod \"authorino-operator-75cc8d5cb7-lf8jx\" (UID: \"70d76a1d-0156-4342-9631-c5ce326197bb\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-lf8jx"
Jul 06 20:32:33.339421 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:32:33.339389    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-lf8jx"
Jul 06 20:32:33.472324 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:32:33.472271    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-lf8jx"]
Jul 06 20:32:33.474515 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:32:33.474489    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70d76a1d_0156_4342_9631_c5ce326197bb.slice/crio-4c80cc77c4e0fb592281c3662c59aed409c12812e15cfde796fa3e4368a7f396 WatchSource:0}: Error finding container 4c80cc77c4e0fb592281c3662c59aed409c12812e15cfde796fa3e4368a7f396: Status 404 returned error can't find the container with id 4c80cc77c4e0fb592281c3662c59aed409c12812e15cfde796fa3e4368a7f396
Jul 06 20:32:33.940147 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:32:33.940068    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:32:34.183591 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:32:34.183549    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-lf8jx" event={"ID":"70d76a1d-0156-4342-9631-c5ce326197bb","Type":"ContainerStarted","Data":"4c80cc77c4e0fb592281c3662c59aed409c12812e15cfde796fa3e4368a7f396"}
Jul 06 20:32:35.763484 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:32:35.763331    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:32:36.195171 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:32:36.195137    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-lf8jx" event={"ID":"70d76a1d-0156-4342-9631-c5ce326197bb","Type":"ContainerStarted","Data":"546fe0d86c4c7bd4b815963e1d90d33ce6155036ec495d869a9fc2b1e2a3a0da"}
Jul 06 20:32:36.195352 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:32:36.195216    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-lf8jx"
Jul 06 20:32:36.217463 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:32:36.217417    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-lf8jx" podStartSLOduration=2.257049466 podStartE2EDuration="4.217402615s" totalImagesPullingTime="1.960353149s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:32:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:32:36.214931522 +0000 UTC m=+519.931042185" watchObservedRunningTime="2026-07-06 20:32:36.217402615 +0000 UTC m=+519.933513276"
Jul 06 20:32:37.298310 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:32:37.298278    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:32:37.397468 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:32:37.397429    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:32:37.762612 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:32:37.762573    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:32:38.480213 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:32:38.480183    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:32:41.093336 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:32:41.093294    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:32:41.155713 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:32:41.155672    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:32:47.202677 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:32:47.202652    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-lf8jx"
Jul 06 20:32:50.489398 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:32:50.489304    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:32:50.590061 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:32:50.590029    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:32:50.701352 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:32:50.701320    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:32:50.804347 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:32:50.804308    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:32:53.360084 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:32:53.360051    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:32:53.468512 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:32:53.468472    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:32:55.901471 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:32:55.901441    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:32:56.003705 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:32:56.003675    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:32:58.467592 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:32:58.467561    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:32:59.508705 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:32:59.508668    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:32:59.570285 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:32:59.570253    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:33:09.101693 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:33:09.101658    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:33:09.202116 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:33:09.202080    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:33:09.493053 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:33:09.492957    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:33:09.555887 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:33:09.555855    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:33:14.928914 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:33:14.928876    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:33:15.024569 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:33:15.024531    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:33:19.441952 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:33:19.441920    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:33:27.930691 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:33:27.930656    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:33:28.098382 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:33:28.098345    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:33:30.039865 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:33:30.039830    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:33:30.140782 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:33:30.140751    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:33:30.223894 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:33:30.223861    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:33:30.465205 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:33:30.465171    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-clwd5"]
Jul 06 20:33:30.467981 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:33:30.467963    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-clwd5"
Jul 06 20:33:30.470900 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:33:30.470879    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-c8bz4\""
Jul 06 20:33:30.471024 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:33:30.470880    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 06 20:33:30.482852 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:33:30.482828    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-clwd5"]
Jul 06 20:33:30.498857 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:33:30.498831    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-clwd5"]
Jul 06 20:33:30.571240 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:33:30.571205    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4lmd\" (UniqueName: \"kubernetes.io/projected/f0f20c5b-abd5-41f7-bca1-24dc1b5bc457-kube-api-access-c4lmd\") pod \"limitador-limitador-54d6d988d5-clwd5\" (UID: \"f0f20c5b-abd5-41f7-bca1-24dc1b5bc457\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-clwd5"
Jul 06 20:33:30.571406 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:33:30.571268    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/f0f20c5b-abd5-41f7-bca1-24dc1b5bc457-config-file\") pod \"limitador-limitador-54d6d988d5-clwd5\" (UID: \"f0f20c5b-abd5-41f7-bca1-24dc1b5bc457\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-clwd5"
Jul 06 20:33:30.671964 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:33:30.671926    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4lmd\" (UniqueName: \"kubernetes.io/projected/f0f20c5b-abd5-41f7-bca1-24dc1b5bc457-kube-api-access-c4lmd\") pod \"limitador-limitador-54d6d988d5-clwd5\" (UID: \"f0f20c5b-abd5-41f7-bca1-24dc1b5bc457\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-clwd5"
Jul 06 20:33:30.672149 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:33:30.671987    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/f0f20c5b-abd5-41f7-bca1-24dc1b5bc457-config-file\") pod \"limitador-limitador-54d6d988d5-clwd5\" (UID: \"f0f20c5b-abd5-41f7-bca1-24dc1b5bc457\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-clwd5"
Jul 06 20:33:30.672593 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:33:30.672575    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/f0f20c5b-abd5-41f7-bca1-24dc1b5bc457-config-file\") pod \"limitador-limitador-54d6d988d5-clwd5\" (UID: \"f0f20c5b-abd5-41f7-bca1-24dc1b5bc457\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-clwd5"
Jul 06 20:33:30.681100 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:33:30.681075    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4lmd\" (UniqueName: \"kubernetes.io/projected/f0f20c5b-abd5-41f7-bca1-24dc1b5bc457-kube-api-access-c4lmd\") pod \"limitador-limitador-54d6d988d5-clwd5\" (UID: \"f0f20c5b-abd5-41f7-bca1-24dc1b5bc457\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-clwd5"
Jul 06 20:33:30.777808 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:33:30.777694    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-clwd5"
Jul 06 20:33:30.915389 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:33:30.915314    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:33:30.920535 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:33:30.917505    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-clwd5"]
Jul 06 20:33:31.013662 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:33:31.013619    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:33:31.448574 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:33:31.448384    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-clwd5" event={"ID":"f0f20c5b-abd5-41f7-bca1-24dc1b5bc457","Type":"ContainerStarted","Data":"8feeb33a58614767afd19aab2e84f42a877f5a904233884afff02d7250b17c2b"}
Jul 06 20:33:33.459711 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:33:33.459683    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-clwd5" event={"ID":"f0f20c5b-abd5-41f7-bca1-24dc1b5bc457","Type":"ContainerStarted","Data":"e19eec68e8c708615aa4a8d125a7ca3c6e0964eedbfaf4fdfb4f3436f5f9dcdb"}
Jul 06 20:33:33.460145 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:33:33.459881    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-clwd5"
Jul 06 20:33:33.479389 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:33:33.479346    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-clwd5" podStartSLOduration=1.035126549 podStartE2EDuration="3.479332856s" totalImagesPullingTime="2.444206307s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:33:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:33:33.477509759 +0000 UTC m=+577.193620420" watchObservedRunningTime="2026-07-06 20:33:33.479332856 +0000 UTC m=+577.195443516"
Jul 06 20:33:44.465752 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:33:44.465693    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-clwd5"
Jul 06 20:33:46.660533 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:33:46.660494    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:33:56.688984 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:33:56.688956    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-2m7gh_bb2a41aa-a9fc-4ec5-9b78-5e22c1fd301b/console-operator/1.log"
Jul 06 20:33:56.693611 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:33:56.693579    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-2m7gh_bb2a41aa-a9fc-4ec5-9b78-5e22c1fd301b/console-operator/1.log"
Jul 06 20:33:59.075179 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:33:59.075143    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:33:59.421369 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:33:59.421284    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:33:59.605559 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:33:59.605527    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-764454fd8-qdv57"]
Jul 06 20:33:59.609132 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:33:59.609113    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-764454fd8-qdv57"
Jul 06 20:33:59.612557 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:33:59.612504    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-pwjkt\""
Jul 06 20:33:59.618700 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:33:59.618679    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-764454fd8-qdv57"]
Jul 06 20:33:59.740138 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:33:59.740054    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7c7d4\" (UniqueName: \"kubernetes.io/projected/c604c88e-11ab-4e58-a129-1960a1c7af94-kube-api-access-7c7d4\") pod \"authorino-764454fd8-qdv57\" (UID: \"c604c88e-11ab-4e58-a129-1960a1c7af94\") " pod="kuadrant-system/authorino-764454fd8-qdv57"
Jul 06 20:33:59.764622 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:33:59.764595    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-764454fd8-qdv57"]
Jul 06 20:33:59.764929 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:33:59.764898    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-7c7d4], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-764454fd8-qdv57" podUID="c604c88e-11ab-4e58-a129-1960a1c7af94"
Jul 06 20:33:59.796796 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:33:59.796761    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6c9df6cfbd-6458d"]
Jul 06 20:33:59.800159 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:33:59.800135    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6c9df6cfbd-6458d"
Jul 06 20:33:59.802659 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:33:59.802631    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Jul 06 20:33:59.810037 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:33:59.810005    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6c9df6cfbd-6458d"]
Jul 06 20:33:59.841141 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:33:59.841110    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7c7d4\" (UniqueName: \"kubernetes.io/projected/c604c88e-11ab-4e58-a129-1960a1c7af94-kube-api-access-7c7d4\") pod \"authorino-764454fd8-qdv57\" (UID: \"c604c88e-11ab-4e58-a129-1960a1c7af94\") " pod="kuadrant-system/authorino-764454fd8-qdv57"
Jul 06 20:33:59.849958 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:33:59.849931    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7c7d4\" (UniqueName: \"kubernetes.io/projected/c604c88e-11ab-4e58-a129-1960a1c7af94-kube-api-access-7c7d4\") pod \"authorino-764454fd8-qdv57\" (UID: \"c604c88e-11ab-4e58-a129-1960a1c7af94\") " pod="kuadrant-system/authorino-764454fd8-qdv57"
Jul 06 20:33:59.941851 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:33:59.941816    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbqqm\" (UniqueName: \"kubernetes.io/projected/d7deda67-00f7-4777-afb2-5c31c1a3b4f3-kube-api-access-gbqqm\") pod \"authorino-6c9df6cfbd-6458d\" (UID: \"d7deda67-00f7-4777-afb2-5c31c1a3b4f3\") " pod="kuadrant-system/authorino-6c9df6cfbd-6458d"
Jul 06 20:33:59.942041 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:33:59.941861    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d7deda67-00f7-4777-afb2-5c31c1a3b4f3-tls-cert\") pod \"authorino-6c9df6cfbd-6458d\" (UID: \"d7deda67-00f7-4777-afb2-5c31c1a3b4f3\") " pod="kuadrant-system/authorino-6c9df6cfbd-6458d"
Jul 06 20:34:00.042589 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:34:00.042512    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d7deda67-00f7-4777-afb2-5c31c1a3b4f3-tls-cert\") pod \"authorino-6c9df6cfbd-6458d\" (UID: \"d7deda67-00f7-4777-afb2-5c31c1a3b4f3\") " pod="kuadrant-system/authorino-6c9df6cfbd-6458d"
Jul 06 20:34:00.042774 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:34:00.042604    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbqqm\" (UniqueName: \"kubernetes.io/projected/d7deda67-00f7-4777-afb2-5c31c1a3b4f3-kube-api-access-gbqqm\") pod \"authorino-6c9df6cfbd-6458d\" (UID: \"d7deda67-00f7-4777-afb2-5c31c1a3b4f3\") " pod="kuadrant-system/authorino-6c9df6cfbd-6458d"
Jul 06 20:34:00.044987 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:34:00.044964    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d7deda67-00f7-4777-afb2-5c31c1a3b4f3-tls-cert\") pod \"authorino-6c9df6cfbd-6458d\" (UID: \"d7deda67-00f7-4777-afb2-5c31c1a3b4f3\") " pod="kuadrant-system/authorino-6c9df6cfbd-6458d"
Jul 06 20:34:00.050471 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:34:00.050452    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbqqm\" (UniqueName: \"kubernetes.io/projected/d7deda67-00f7-4777-afb2-5c31c1a3b4f3-kube-api-access-gbqqm\") pod \"authorino-6c9df6cfbd-6458d\" (UID: \"d7deda67-00f7-4777-afb2-5c31c1a3b4f3\") " pod="kuadrant-system/authorino-6c9df6cfbd-6458d"
Jul 06 20:34:00.112785 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:34:00.112753    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6c9df6cfbd-6458d"
Jul 06 20:34:00.245465 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:34:00.245440    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6c9df6cfbd-6458d"]
Jul 06 20:34:00.247659 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:34:00.247630    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd7deda67_00f7_4777_afb2_5c31c1a3b4f3.slice/crio-2caa41922cc1f5af885efce1078993cf804ff85cf5be46c3ba124138a757fd14 WatchSource:0}: Error finding container 2caa41922cc1f5af885efce1078993cf804ff85cf5be46c3ba124138a757fd14: Status 404 returned error can't find the container with id 2caa41922cc1f5af885efce1078993cf804ff85cf5be46c3ba124138a757fd14
Jul 06 20:34:00.580914 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:34:00.580877    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6c9df6cfbd-6458d" event={"ID":"d7deda67-00f7-4777-afb2-5c31c1a3b4f3","Type":"ContainerStarted","Data":"2caa41922cc1f5af885efce1078993cf804ff85cf5be46c3ba124138a757fd14"}
Jul 06 20:34:00.580914 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:34:00.580902    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-764454fd8-qdv57"
Jul 06 20:34:00.589448 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:34:00.589422    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-764454fd8-qdv57"
Jul 06 20:34:00.748856 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:34:00.748817    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7c7d4\" (UniqueName: \"kubernetes.io/projected/c604c88e-11ab-4e58-a129-1960a1c7af94-kube-api-access-7c7d4\") pod \"c604c88e-11ab-4e58-a129-1960a1c7af94\" (UID: \"c604c88e-11ab-4e58-a129-1960a1c7af94\") "
Jul 06 20:34:00.750865 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:34:00.750840    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c604c88e-11ab-4e58-a129-1960a1c7af94-kube-api-access-7c7d4" (OuterVolumeSpecName: "kube-api-access-7c7d4") pod "c604c88e-11ab-4e58-a129-1960a1c7af94" (UID: "c604c88e-11ab-4e58-a129-1960a1c7af94"). InnerVolumeSpecName "kube-api-access-7c7d4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 20:34:00.850157 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:34:00.850127    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7c7d4\" (UniqueName: \"kubernetes.io/projected/c604c88e-11ab-4e58-a129-1960a1c7af94-kube-api-access-7c7d4\") on node \"ip-10-0-141-123.ec2.internal\" DevicePath \"\""
Jul 06 20:34:01.586943 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:34:01.586883    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-764454fd8-qdv57"
Jul 06 20:34:01.614577 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:34:01.614502    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-764454fd8-qdv57"]
Jul 06 20:34:01.619027 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:34:01.618944    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-764454fd8-qdv57"]
Jul 06 20:34:02.440585 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:34:02.440552    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:34:02.834143 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:34:02.834109    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c604c88e-11ab-4e58-a129-1960a1c7af94" path="/var/lib/kubelet/pods/c604c88e-11ab-4e58-a129-1960a1c7af94/volumes"
Jul 06 20:34:03.597471 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:34:03.597383    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6c9df6cfbd-6458d" event={"ID":"d7deda67-00f7-4777-afb2-5c31c1a3b4f3","Type":"ContainerStarted","Data":"27404418b266a2a5ab00f0353403c4cd1ea6919b4b1b13a5803ff54e6c7aaf84"}
Jul 06 20:34:03.626906 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:34:03.626843    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-6c9df6cfbd-6458d" podStartSLOduration=1.5589833629999998 podStartE2EDuration="4.626826912s" totalImagesPullingTime="3.067843549s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:33:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:34:03.624899278 +0000 UTC m=+607.341009958" watchObservedRunningTime="2026-07-06 20:34:03.626826912 +0000 UTC m=+607.342937590"
Jul 06 20:34:08.402693 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:34:08.402655    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:34:08.492129 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:34:08.492099    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:34:08.592283 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:34:08.592247    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:34:11.237690 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:34:11.237644    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:34:11.414039 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:34:11.414003    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:34:44.229383 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:34:44.229343    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:34:44.624721 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:34:44.624683    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:34:45.023088 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:34:45.023003    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:34:45.120763 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:34:45.120706    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:34:45.189921 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:34:45.189891    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:34:45.310464 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:34:45.310428    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-clwd5"]
Jul 06 20:34:55.012801 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:34:55.012768    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:34:56.233040 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:34:56.233005    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:34:56.649088 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:34:56.649057    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:34:56.749297 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:34:56.749263    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:35:01.067136 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:35:01.067095    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:35:01.647918 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:35:01.647882    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-clwd5"]
Jul 06 20:35:12.347808 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:35:12.347778    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:35:12.788721 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:35:12.788627    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-clwd5"]
Jul 06 20:35:16.763596 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:35:16.763561    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:35:17.880789 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:35:17.880750    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-clwd5"]
Jul 06 20:35:22.594348 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:35:22.594308    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:35:24.111324 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:35:24.111290    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:35:32.825773 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:35:32.825682    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:35:34.394761 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:35:34.394712    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-clwd5"]
Jul 06 20:35:43.135621 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:35:43.135586    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:35:44.988860 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:35:44.988825    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-clwd5"]
Jul 06 20:36:00.027001 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:36:00.026930    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-78mx7"]
Jul 06 20:36:00.029486 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:36:00.029466    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-78mx7"
Jul 06 20:36:00.033176 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:36:00.033156    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 06 20:36:00.033830 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:36:00.033813    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-6dg98\""
Jul 06 20:36:00.033896 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:36:00.033815    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 06 20:36:00.034126 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:36:00.034112    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Jul 06 20:36:00.044282 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:36:00.044261    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-78mx7"]
Jul 06 20:36:00.077463 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:36:00.077435    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:00.092853 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:36:00.092809    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a019d9e3-7fa0-4ed9-b96d-eda20e20f798-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-78mx7\" (UID: \"a019d9e3-7fa0-4ed9-b96d-eda20e20f798\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-78mx7"
Jul 06 20:36:00.092977 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:36:00.092868    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a019d9e3-7fa0-4ed9-b96d-eda20e20f798-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-78mx7\" (UID: \"a019d9e3-7fa0-4ed9-b96d-eda20e20f798\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-78mx7"
Jul 06 20:36:00.092977 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:36:00.092915    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a019d9e3-7fa0-4ed9-b96d-eda20e20f798-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-78mx7\" (UID: \"a019d9e3-7fa0-4ed9-b96d-eda20e20f798\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-78mx7"
Jul 06 20:36:00.093096 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:36:00.093010    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a019d9e3-7fa0-4ed9-b96d-eda20e20f798-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-78mx7\" (UID: \"a019d9e3-7fa0-4ed9-b96d-eda20e20f798\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-78mx7"
Jul 06 20:36:00.093096 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:36:00.093060    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6p97\" (UniqueName: \"kubernetes.io/projected/a019d9e3-7fa0-4ed9-b96d-eda20e20f798-kube-api-access-l6p97\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-78mx7\" (UID: \"a019d9e3-7fa0-4ed9-b96d-eda20e20f798\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-78mx7"
Jul 06 20:36:00.093195 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:36:00.093101    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a019d9e3-7fa0-4ed9-b96d-eda20e20f798-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-78mx7\" (UID: \"a019d9e3-7fa0-4ed9-b96d-eda20e20f798\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-78mx7"
Jul 06 20:36:00.194122 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:36:00.194086    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a019d9e3-7fa0-4ed9-b96d-eda20e20f798-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-78mx7\" (UID: \"a019d9e3-7fa0-4ed9-b96d-eda20e20f798\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-78mx7"
Jul 06 20:36:00.194326 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:36:00.194134    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6p97\" (UniqueName: \"kubernetes.io/projected/a019d9e3-7fa0-4ed9-b96d-eda20e20f798-kube-api-access-l6p97\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-78mx7\" (UID: \"a019d9e3-7fa0-4ed9-b96d-eda20e20f798\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-78mx7"
Jul 06 20:36:00.194326 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:36:00.194162    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a019d9e3-7fa0-4ed9-b96d-eda20e20f798-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-78mx7\" (UID: \"a019d9e3-7fa0-4ed9-b96d-eda20e20f798\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-78mx7"
Jul 06 20:36:00.194326 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:36:00.194229    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a019d9e3-7fa0-4ed9-b96d-eda20e20f798-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-78mx7\" (UID: \"a019d9e3-7fa0-4ed9-b96d-eda20e20f798\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-78mx7"
Jul 06 20:36:00.194326 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:36:00.194259    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a019d9e3-7fa0-4ed9-b96d-eda20e20f798-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-78mx7\" (UID: \"a019d9e3-7fa0-4ed9-b96d-eda20e20f798\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-78mx7"
Jul 06 20:36:00.194326 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:36:00.194281    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a019d9e3-7fa0-4ed9-b96d-eda20e20f798-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-78mx7\" (UID: \"a019d9e3-7fa0-4ed9-b96d-eda20e20f798\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-78mx7"
Jul 06 20:36:00.196345 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:36:00.194829    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a019d9e3-7fa0-4ed9-b96d-eda20e20f798-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-78mx7\" (UID: \"a019d9e3-7fa0-4ed9-b96d-eda20e20f798\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-78mx7"
Jul 06 20:36:00.196345 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:36:00.194849    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a019d9e3-7fa0-4ed9-b96d-eda20e20f798-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-78mx7\" (UID: \"a019d9e3-7fa0-4ed9-b96d-eda20e20f798\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-78mx7"
Jul 06 20:36:00.196345 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:36:00.195001    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a019d9e3-7fa0-4ed9-b96d-eda20e20f798-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-78mx7\" (UID: \"a019d9e3-7fa0-4ed9-b96d-eda20e20f798\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-78mx7"
Jul 06 20:36:00.202307 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:36:00.202280    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a019d9e3-7fa0-4ed9-b96d-eda20e20f798-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-78mx7\" (UID: \"a019d9e3-7fa0-4ed9-b96d-eda20e20f798\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-78mx7"
Jul 06 20:36:00.202555 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:36:00.202537    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a019d9e3-7fa0-4ed9-b96d-eda20e20f798-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-78mx7\" (UID: \"a019d9e3-7fa0-4ed9-b96d-eda20e20f798\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-78mx7"
Jul 06 20:36:00.204491 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:36:00.204471    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6p97\" (UniqueName: \"kubernetes.io/projected/a019d9e3-7fa0-4ed9-b96d-eda20e20f798-kube-api-access-l6p97\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-78mx7\" (UID: \"a019d9e3-7fa0-4ed9-b96d-eda20e20f798\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-78mx7"
Jul 06 20:36:00.338949 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:36:00.338918    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-78mx7"
Jul 06 20:36:00.470984 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:36:00.470957    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-78mx7"]
Jul 06 20:36:00.476181 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:36:00.475999    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda019d9e3_7fa0_4ed9_b96d_eda20e20f798.slice/crio-36754af2d03e8e1023103ef02119f86e418936c4f6e62da20092e41982f31cd9 WatchSource:0}: Error finding container 36754af2d03e8e1023103ef02119f86e418936c4f6e62da20092e41982f31cd9: Status 404 returned error can't find the container with id 36754af2d03e8e1023103ef02119f86e418936c4f6e62da20092e41982f31cd9
Jul 06 20:36:00.477033 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:36:00.477015    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 06 20:36:01.142306 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:36:01.142273    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-78mx7" event={"ID":"a019d9e3-7fa0-4ed9-b96d-eda20e20f798","Type":"ContainerStarted","Data":"36754af2d03e8e1023103ef02119f86e418936c4f6e62da20092e41982f31cd9"}
Jul 06 20:36:02.148474 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:36:02.148436    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-78mx7" event={"ID":"a019d9e3-7fa0-4ed9-b96d-eda20e20f798","Type":"ContainerStarted","Data":"2da36ee32ca5f589d5b9bc692bb2c32f21568c736174fd6f6b70e44cf02722bb"}
Jul 06 20:36:02.168256 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:36:02.168210    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-78mx7" podStartSLOduration=1.1818096869999999 podStartE2EDuration="2.168195576s" totalImagesPullingTime="986.385889ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:36:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:36:02.16617238 +0000 UTC m=+725.882283050" watchObservedRunningTime="2026-07-06 20:36:02.168195576 +0000 UTC m=+725.884306238"
Jul 06 20:36:02.792519 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:36:02.792482    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-clwd5"]
Jul 06 20:36:10.339860 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:36:10.339818    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-78mx7"
Jul 06 20:36:10.340259 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:36:10.339902    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-78mx7"
Jul 06 20:36:10.352480 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:36:10.352456    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-78mx7"
Jul 06 20:36:11.198496 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:36:11.198464    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-78mx7"
Jul 06 20:37:05.797387 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:37:05.797355    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:37:05.898699 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:37:05.898664    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:37:29.541694 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:37:29.541599    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-74564584cc-w8nz7"]
Jul 06 20:37:29.545387 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:37:29.545363    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-74564584cc-w8nz7"
Jul 06 20:37:29.554788 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:37:29.554760    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-74564584cc-w8nz7"]
Jul 06 20:37:29.607763 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:37:29.607694    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l86wq\" (UniqueName: \"kubernetes.io/projected/8903f259-39c4-4473-954c-3075c773278b-kube-api-access-l86wq\") pod \"authorino-74564584cc-w8nz7\" (UID: \"8903f259-39c4-4473-954c-3075c773278b\") " pod="kuadrant-system/authorino-74564584cc-w8nz7"
Jul 06 20:37:29.607973 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:37:29.607837    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8903f259-39c4-4473-954c-3075c773278b-tls-cert\") pod \"authorino-74564584cc-w8nz7\" (UID: \"8903f259-39c4-4473-954c-3075c773278b\") " pod="kuadrant-system/authorino-74564584cc-w8nz7"
Jul 06 20:37:29.708368 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:37:29.708330    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8903f259-39c4-4473-954c-3075c773278b-tls-cert\") pod \"authorino-74564584cc-w8nz7\" (UID: \"8903f259-39c4-4473-954c-3075c773278b\") " pod="kuadrant-system/authorino-74564584cc-w8nz7"
Jul 06 20:37:29.708576 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:37:29.708417    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l86wq\" (UniqueName: \"kubernetes.io/projected/8903f259-39c4-4473-954c-3075c773278b-kube-api-access-l86wq\") pod \"authorino-74564584cc-w8nz7\" (UID: \"8903f259-39c4-4473-954c-3075c773278b\") " pod="kuadrant-system/authorino-74564584cc-w8nz7"
Jul 06 20:37:29.710844 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:37:29.710820    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8903f259-39c4-4473-954c-3075c773278b-tls-cert\") pod \"authorino-74564584cc-w8nz7\" (UID: \"8903f259-39c4-4473-954c-3075c773278b\") " pod="kuadrant-system/authorino-74564584cc-w8nz7"
Jul 06 20:37:29.717086 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:37:29.717058    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l86wq\" (UniqueName: \"kubernetes.io/projected/8903f259-39c4-4473-954c-3075c773278b-kube-api-access-l86wq\") pod \"authorino-74564584cc-w8nz7\" (UID: \"8903f259-39c4-4473-954c-3075c773278b\") " pod="kuadrant-system/authorino-74564584cc-w8nz7"
Jul 06 20:37:29.856172 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:37:29.856138    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-74564584cc-w8nz7"
Jul 06 20:37:29.991202 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:37:29.991178    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-74564584cc-w8nz7"]
Jul 06 20:37:29.993833 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:37:29.993803    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8903f259_39c4_4473_954c_3075c773278b.slice/crio-2c88136fb1ee848248a9e1c29135b1a912764f9b4ad1956083151f3f5297b68d WatchSource:0}: Error finding container 2c88136fb1ee848248a9e1c29135b1a912764f9b4ad1956083151f3f5297b68d: Status 404 returned error can't find the container with id 2c88136fb1ee848248a9e1c29135b1a912764f9b4ad1956083151f3f5297b68d
Jul 06 20:37:30.558425 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:37:30.558388    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-74564584cc-w8nz7" event={"ID":"8903f259-39c4-4473-954c-3075c773278b","Type":"ContainerStarted","Data":"36f67183b74b525c78be57d7134048cef41d300c13df3a954a8b11e596b3838a"}
Jul 06 20:37:30.559002 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:37:30.558434    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-74564584cc-w8nz7" event={"ID":"8903f259-39c4-4473-954c-3075c773278b","Type":"ContainerStarted","Data":"2c88136fb1ee848248a9e1c29135b1a912764f9b4ad1956083151f3f5297b68d"}
Jul 06 20:37:30.583572 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:37:30.583506    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-74564584cc-w8nz7" podStartSLOduration=1.164392367 podStartE2EDuration="1.583481707s" totalImagesPullingTime="419.08934ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:37:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:37:30.580901641 +0000 UTC m=+814.297012305" watchObservedRunningTime="2026-07-06 20:37:30.583481707 +0000 UTC m=+814.299592371"
Jul 06 20:37:30.618303 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:37:30.618277    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6c9df6cfbd-6458d"]
Jul 06 20:37:30.618507 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:37:30.618463    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-6c9df6cfbd-6458d" podUID="d7deda67-00f7-4777-afb2-5c31c1a3b4f3" containerName="authorino" containerID="cri-o://27404418b266a2a5ab00f0353403c4cd1ea6919b4b1b13a5803ff54e6c7aaf84" gracePeriod=30
Jul 06 20:37:30.867242 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:37:30.867218    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6c9df6cfbd-6458d"
Jul 06 20:37:30.920276 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:37:30.920243    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d7deda67-00f7-4777-afb2-5c31c1a3b4f3-tls-cert\") pod \"d7deda67-00f7-4777-afb2-5c31c1a3b4f3\" (UID: \"d7deda67-00f7-4777-afb2-5c31c1a3b4f3\") "
Jul 06 20:37:30.920276 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:37:30.920290    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gbqqm\" (UniqueName: \"kubernetes.io/projected/d7deda67-00f7-4777-afb2-5c31c1a3b4f3-kube-api-access-gbqqm\") pod \"d7deda67-00f7-4777-afb2-5c31c1a3b4f3\" (UID: \"d7deda67-00f7-4777-afb2-5c31c1a3b4f3\") "
Jul 06 20:37:30.922589 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:37:30.922552    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7deda67-00f7-4777-afb2-5c31c1a3b4f3-kube-api-access-gbqqm" (OuterVolumeSpecName: "kube-api-access-gbqqm") pod "d7deda67-00f7-4777-afb2-5c31c1a3b4f3" (UID: "d7deda67-00f7-4777-afb2-5c31c1a3b4f3"). InnerVolumeSpecName "kube-api-access-gbqqm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 20:37:30.930955 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:37:30.930926    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7deda67-00f7-4777-afb2-5c31c1a3b4f3-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "d7deda67-00f7-4777-afb2-5c31c1a3b4f3" (UID: "d7deda67-00f7-4777-afb2-5c31c1a3b4f3"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 20:37:31.021529 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:37:31.021440    2563 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d7deda67-00f7-4777-afb2-5c31c1a3b4f3-tls-cert\") on node \"ip-10-0-141-123.ec2.internal\" DevicePath \"\""
Jul 06 20:37:31.021529 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:37:31.021471    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbqqm\" (UniqueName: \"kubernetes.io/projected/d7deda67-00f7-4777-afb2-5c31c1a3b4f3-kube-api-access-gbqqm\") on node \"ip-10-0-141-123.ec2.internal\" DevicePath \"\""
Jul 06 20:37:31.231875 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:37:31.231833    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:37:31.346870 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:37:31.346779    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:37:31.565072 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:37:31.565030    2563 generic.go:356] "Generic (PLEG): container finished" podID="d7deda67-00f7-4777-afb2-5c31c1a3b4f3" containerID="27404418b266a2a5ab00f0353403c4cd1ea6919b4b1b13a5803ff54e6c7aaf84" exitCode=0
Jul 06 20:37:31.565494 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:37:31.565077    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6c9df6cfbd-6458d"
Jul 06 20:37:31.565494 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:37:31.565120    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6c9df6cfbd-6458d" event={"ID":"d7deda67-00f7-4777-afb2-5c31c1a3b4f3","Type":"ContainerDied","Data":"27404418b266a2a5ab00f0353403c4cd1ea6919b4b1b13a5803ff54e6c7aaf84"}
Jul 06 20:37:31.565494 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:37:31.565165    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6c9df6cfbd-6458d" event={"ID":"d7deda67-00f7-4777-afb2-5c31c1a3b4f3","Type":"ContainerDied","Data":"2caa41922cc1f5af885efce1078993cf804ff85cf5be46c3ba124138a757fd14"}
Jul 06 20:37:31.565494 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:37:31.565187    2563 scope.go:117] "RemoveContainer" containerID="27404418b266a2a5ab00f0353403c4cd1ea6919b4b1b13a5803ff54e6c7aaf84"
Jul 06 20:37:31.574579 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:37:31.574537    2563 scope.go:117] "RemoveContainer" containerID="27404418b266a2a5ab00f0353403c4cd1ea6919b4b1b13a5803ff54e6c7aaf84"
Jul 06 20:37:31.574902 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:37:31.574882    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27404418b266a2a5ab00f0353403c4cd1ea6919b4b1b13a5803ff54e6c7aaf84\": container with ID starting with 27404418b266a2a5ab00f0353403c4cd1ea6919b4b1b13a5803ff54e6c7aaf84 not found: ID does not exist" containerID="27404418b266a2a5ab00f0353403c4cd1ea6919b4b1b13a5803ff54e6c7aaf84"
Jul 06 20:37:31.575011 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:37:31.574913    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27404418b266a2a5ab00f0353403c4cd1ea6919b4b1b13a5803ff54e6c7aaf84"} err="failed to get container status \"27404418b266a2a5ab00f0353403c4cd1ea6919b4b1b13a5803ff54e6c7aaf84\": rpc error: code = NotFound desc = could not find container \"27404418b266a2a5ab00f0353403c4cd1ea6919b4b1b13a5803ff54e6c7aaf84\": container with ID starting with 27404418b266a2a5ab00f0353403c4cd1ea6919b4b1b13a5803ff54e6c7aaf84 not found: ID does not exist"
Jul 06 20:37:31.615228 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:37:31.615141    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6c9df6cfbd-6458d"]
Jul 06 20:37:31.623691 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:37:31.623660    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6c9df6cfbd-6458d"]
Jul 06 20:37:32.650884 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:37:32.650842    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:37:32.834064 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:37:32.834027    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d7deda67-00f7-4777-afb2-5c31c1a3b4f3" path="/var/lib/kubelet/pods/d7deda67-00f7-4777-afb2-5c31c1a3b4f3/volumes"
Jul 06 20:37:32.834333 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:37:32.834309    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:38:01.797223 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:38:01.797188    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-clwd5"]
Jul 06 20:38:11.608213 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:38:11.608177    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-clwd5"]
Jul 06 20:38:23.019207 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:38:23.019175    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-clwd5"]
Jul 06 20:38:31.415067 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:38:31.415033    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-clwd5"]
Jul 06 20:38:41.999568 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:38:41.999532    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-clwd5"]
Jul 06 20:38:44.656179 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:38:44.656133    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:38:46.731358 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:38:46.731304    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:38:56.720090 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:38:56.720060    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-2m7gh_bb2a41aa-a9fc-4ec5-9b78-5e22c1fd301b/console-operator/1.log"
Jul 06 20:38:56.721789 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:38:56.721765    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-2m7gh_bb2a41aa-a9fc-4ec5-9b78-5e22c1fd301b/console-operator/1.log"
Jul 06 20:39:10.990076 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:39:10.990040    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-clwd5"]
Jul 06 20:39:21.899387 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:39:21.899354    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-clwd5"]
Jul 06 20:39:27.821860 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:39:27.821813    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:39:27.919946 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:39:27.919910    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:39:28.075207 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:39:28.075119    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:39:33.897918 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:39:33.897883    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:39:34.084876 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:39:34.084834    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:39:34.214582 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:39:34.214474    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:39:34.316101 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:39:34.316067    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:39:40.225724 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:39:40.225690    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:39:40.341664 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:39:40.341627    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:39:56.345682 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:39:56.345644    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:39:57.690287 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:39:57.690254    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-clwd5"]
Jul 06 20:40:06.889492 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:40:06.889451    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-clwd5"]
Jul 06 20:40:15.484750 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:40:15.484702    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-clwd5"]
Jul 06 20:40:20.098240 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:40:20.098140    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:40:20.177552 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:40:20.177515    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:40:26.989782 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:40:26.989742    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-clwd5"]
Jul 06 20:40:36.708607 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:40:36.708576    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-clwd5"]
Jul 06 20:40:52.002718 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:40:52.002676    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-clwd5"]
Jul 06 20:40:58.899927 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:40:58.899891    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-clwd5"]
Jul 06 20:41:07.592517 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:41:07.592481    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-clwd5"]
Jul 06 20:41:29.493392 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:41:29.493357    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-clwd5"]
Jul 06 20:41:41.303532 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:41:41.303498    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-clwd5"]
Jul 06 20:41:59.926352 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:41:59.926265    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-clwd5"]
Jul 06 20:42:08.057905 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:42:08.057870    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:42:08.850810 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:42:08.850773    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:42:21.000949 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:42:21.000914    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-clwd5"]
Jul 06 20:42:41.847197 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:42:41.847152    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-clwd5"]
Jul 06 20:42:53.552239 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:42:53.552200    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-clwd5"]
Jul 06 20:43:04.056527 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:43:04.056487    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-clwd5"]
Jul 06 20:43:13.616171 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:43:13.616135    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-clwd5"]
Jul 06 20:43:23.014806 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:43:23.014710    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-clwd5"]
Jul 06 20:43:32.097258 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:43:32.097224    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-clwd5"]
Jul 06 20:43:54.612396 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:43:54.612363    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:43:56.749716 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:43:56.749685    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-2m7gh_bb2a41aa-a9fc-4ec5-9b78-5e22c1fd301b/console-operator/1.log"
Jul 06 20:43:56.750217 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:43:56.750129    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-2m7gh_bb2a41aa-a9fc-4ec5-9b78-5e22c1fd301b/console-operator/1.log"
Jul 06 20:44:38.086123 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:44:38.086091    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:44:38.196491 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:44:38.196458    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:44:50.238881 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:44:50.238749    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:44:50.369614 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:44:50.369570    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:45:22.289290 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:45:22.289255    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-clwd5"]
Jul 06 20:45:31.388520 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:45:31.388486    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-clwd5"]
Jul 06 20:45:44.003582 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:45:44.003546    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-clwd5"]
Jul 06 20:45:54.894276 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:45:54.894240    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-clwd5"]
Jul 06 20:46:06.018171 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:46:06.018136    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-clwd5"]
Jul 06 20:46:17.386974 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:46:17.386941    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-clwd5"]
Jul 06 20:46:26.205197 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:46:26.205116    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-clwd5"]
Jul 06 20:46:36.695865 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:46:36.695828    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-clwd5"]
Jul 06 20:46:48.499984 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:46:48.499949    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-clwd5"]
Jul 06 20:46:56.787426 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:46:56.787392    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-clwd5"]
Jul 06 20:47:07.098258 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:47:07.098226    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-clwd5"]
Jul 06 20:47:29.099265 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:47:29.099236    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-clwd5"]
Jul 06 20:47:37.899578 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:47:37.899539    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-clwd5"]
Jul 06 20:47:49.489965 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:47:49.489876    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-clwd5"]
Jul 06 20:48:01.587346 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:48:01.587312    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-clwd5"]
Jul 06 20:48:20.283410 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:48:20.283372    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-clwd5"]
Jul 06 20:48:29.677196 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:48:29.677155    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-clwd5"]
Jul 06 20:48:38.689099 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:48:38.689063    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-clwd5"]
Jul 06 20:48:48.298752 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:48:48.298695    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-clwd5"]
Jul 06 20:48:55.443331 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:48:55.443287    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:48:56.778459 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:48:56.778430    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-2m7gh_bb2a41aa-a9fc-4ec5-9b78-5e22c1fd301b/console-operator/1.log"
Jul 06 20:48:56.780391 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:48:56.780368    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-2m7gh_bb2a41aa-a9fc-4ec5-9b78-5e22c1fd301b/console-operator/1.log"
Jul 06 20:48:57.689482 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:48:57.689447    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-clwd5"]
Jul 06 20:49:06.800577 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:49:06.800540    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-clwd5"]
Jul 06 20:49:20.803373 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:49:20.803290    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-clwd5"]
Jul 06 20:49:29.605463 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:49:29.605425    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-clwd5"]
Jul 06 20:49:38.498043 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:49:38.498005    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-clwd5"]
Jul 06 20:49:49.390184 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:49:49.390140    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-clwd5"]
Jul 06 20:50:00.093332 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:50:00.093298    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-clwd5"]
Jul 06 20:50:09.089054 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:50:09.089022    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-clwd5"]
Jul 06 20:50:21.101953 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:50:21.101914    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-clwd5"]
Jul 06 20:50:32.202309 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:50:32.202266    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-clwd5"]
Jul 06 20:50:45.707394 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:50:45.707355    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-clwd5"]
Jul 06 20:50:49.660079 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:50:49.659999    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:50:54.204404 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:50:54.204371    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-clwd5"]
Jul 06 20:50:58.431225 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:50:58.431193    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:51:06.591865 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:51:06.591833    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-clwd5"]
Jul 06 20:51:06.790548 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:51:06.790516    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:51:11.926492 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:51:11.926458    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:51:14.898943 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:51:14.898904    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-clwd5"]
Jul 06 20:51:14.990564 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:51:14.990529    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:51:15.090710 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:51:15.090674    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:51:15.504880 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:51:15.504843    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:51:21.591887 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:51:21.591849    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:51:21.712600 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:51:21.712563    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:51:21.985590 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:51:21.985551    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:51:22.154477 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:51:22.154395    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:51:23.070145 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:51:23.070112    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:51:23.180087 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:51:23.180052    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:51:23.511672 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:51:23.511595    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:51:23.853053 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:51:23.853013    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:51:23.952001 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:51:23.951971    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:51:24.920528 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:51:24.920488    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-clwd5"]
Jul 06 20:51:25.906837 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:51:25.906805    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:51:28.170178 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:51:28.170141    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:51:28.394999 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:51:28.394964    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:51:28.472415 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:51:28.472334    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:51:28.694306 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:51:28.694275    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:51:29.705996 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:51:29.705961    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:51:34.374150 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:51:34.374116    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:51:34.472528 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:51:34.472498    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:51:34.681855 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:51:34.681772    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:51:34.881744 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:51:34.881700    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:51:35.761907 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:51:35.761875    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:51:35.859476 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:51:35.859443    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:51:35.969707 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:51:35.969668    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:51:36.431638 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:51:36.431604    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:51:36.499215 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:51:36.499185    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-clwd5"]
Jul 06 20:51:36.503255 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:51:36.503228    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:51:36.583658 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:51:36.583626    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:51:37.780641 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:51:37.780602    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:51:40.874181 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:51:40.874151    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:51:41.214513 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:51:41.214432    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:51:47.169581 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:51:47.169549    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:51:47.339723 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:51:47.339690    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:51:53.100007 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:51:53.099968    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:51:53.200045 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:51:53.200009    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:51:53.335534 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:51:53.335503    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:51:54.460223 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:51:54.460188    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:51:54.567393 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:51:54.567363    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:51:54.636971 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:51:54.636935    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:51:54.976620 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:51:54.976583    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:51:55.219004 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:51:55.218970    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:51:55.320017 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:51:55.319984    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:51:57.093558 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:51:57.093528    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-clwd5"]
Jul 06 20:51:57.705985 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:51:57.705953    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:51:58.907788 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:51:58.907748    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:51:59.098353 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:51:59.098319    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:51:59.201902 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:51:59.201823    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:51:59.273951 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:51:59.273917    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:52:00.668242 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:52:00.668207    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:52:05.382423 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:52:05.382387    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:52:05.462138 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:52:05.462107    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:52:05.788811 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:52:05.788699    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-clwd5"]
Jul 06 20:52:05.909703 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:52:05.909653    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:52:06.017441 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:52:06.017408    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:52:06.110074 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:52:06.110045    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:52:07.588047 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:52:07.588012    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:52:14.108196 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:52:14.108161    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:52:14.220756 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:52:14.220698    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:52:14.351559 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:52:14.351522    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:52:14.448524 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:52:14.448447    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:52:14.883237 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:52:14.883203    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:52:29.742470 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:52:29.742388    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:52:40.200346 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:52:40.200309    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:52:40.878767 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:52:40.878710    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:52:46.780567 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:52:46.780534    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:52:46.895911 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:52:46.895876    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:52:46.986828 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:52:46.986798    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:53:02.907155 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:53:02.907119    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:53:08.845535 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:53:08.845502    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:53:08.914911 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:53:08.914881    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:53:09.023318 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:53:09.023292    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:53:09.116180 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:53:09.116091    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:53:19.581982 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:53:19.581948    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:53:20.389539 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:53:20.389508    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:53:26.651279 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:53:26.651243    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:53:26.755785 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:53:26.755756    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:53:26.893009 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:53:26.892977    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:53:26.991006 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:53:26.990933    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:53:27.817310 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:53:27.817278    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:53:27.946716 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:53:27.946684    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:53:29.187921 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:53:29.187885    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-clwd5"]
Jul 06 20:53:29.617021 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:53:29.616988    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:53:35.589061 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:53:35.589028    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:53:35.685273 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:53:35.685243    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:53:36.247153 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:53:36.247116    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:53:36.487748 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:53:36.487706    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-clwd5"]
Jul 06 20:53:42.369905 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:53:42.369872    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:53:42.486273 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:53:42.486243    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:53:42.610775 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:53:42.610722    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:53:42.957001 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:53:42.956971    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:53:43.692479 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:53:43.692450    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:53:44.689412 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:53:44.689384    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-clwd5"]
Jul 06 20:53:45.878585 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:53:45.878552    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:53:45.978432 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:53:45.978401    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:53:46.199601 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:53:46.199516    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:53:46.575013 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:53:46.574981    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:53:46.850670 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:53:46.850581    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:53:46.949830 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:53:46.949803    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:53:50.311578 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:53:50.311503    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:53:51.115602 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:53:51.115560    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:53:51.264535 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:53:51.264506    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:53:51.512002 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:53:51.511904    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:53:54.702073 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:53:54.702031    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-clwd5"]
Jul 06 20:53:56.805656 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:53:56.805628    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-2m7gh_bb2a41aa-a9fc-4ec5-9b78-5e22c1fd301b/console-operator/1.log"
Jul 06 20:53:56.808103 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:53:56.808083    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-2m7gh_bb2a41aa-a9fc-4ec5-9b78-5e22c1fd301b/console-operator/1.log"
Jul 06 20:53:57.364280 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:53:57.364248    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:53:58.621412 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:53:58.621360    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:54:02.279836 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:54:02.279801    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-clwd5"]
Jul 06 20:54:04.417096 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:54:04.417066    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:54:04.480344 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:54:04.480316    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:54:04.713213 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:54:04.713176    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:54:04.947299 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:54:04.947269    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:54:05.079887 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:54:05.079857    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:54:05.176760 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:54:05.176687    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:54:05.301791 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:54:05.301760    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:54:10.173206 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:54:10.173177    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:54:10.274429 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:54:10.274390    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:54:11.638942 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:54:11.638913    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:54:11.745868 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:54:11.745840    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:54:11.840608 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:54:11.840567    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:54:11.978879 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:54:11.978795    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:54:17.819857 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:54:17.819825    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:54:17.911770 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:54:17.911740    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:54:24.761720 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:54:24.761686    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:54:25.079845 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:54:25.079806    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:54:25.233578 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:54:25.233537    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:54:27.996078 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:54:27.996024    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:54:28.077485 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:54:28.077457    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:54:28.141869 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:54:28.141836    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:54:28.573922 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:54:28.573890    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:54:28.693892 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:54:28.693861    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:54:32.585530 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:54:32.585491    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:54:34.205218 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:54:34.205181    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-clwd5"]
Jul 06 20:54:41.728841 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:54:41.728804    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-1726c2fe/e2e-lifecycle-model-1726c2fe-kserve-66c7cbcbc5-2w8gh"]
Jul 06 20:54:41.729396 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:54:41.729325    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7deda67-00f7-4777-afb2-5c31c1a3b4f3" containerName="authorino"
Jul 06 20:54:41.729396 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:54:41.729344    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7deda67-00f7-4777-afb2-5c31c1a3b4f3" containerName="authorino"
Jul 06 20:54:41.729521 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:54:41.729447    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d7deda67-00f7-4777-afb2-5c31c1a3b4f3" containerName="authorino"
Jul 06 20:54:41.732393 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:54:41.732374    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-1726c2fe/e2e-lifecycle-model-1726c2fe-kserve-66c7cbcbc5-2w8gh"
Jul 06 20:54:41.735594 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:54:41.735574    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-1726c2fe\"/\"e2e-lifecycle-model-1726c2fe-kserve-self-signed-certs\""
Jul 06 20:54:41.735594 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:54:41.735591    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-1726c2fe\"/\"default-dockercfg-4wjps\""
Jul 06 20:54:41.735822 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:54:41.735619    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-1726c2fe\"/\"openshift-service-ca.crt\""
Jul 06 20:54:41.735822 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:54:41.735639    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-1726c2fe\"/\"kube-root-ca.crt\""
Jul 06 20:54:41.743022 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:54:41.743002    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-1726c2fe/e2e-lifecycle-model-1726c2fe-kserve-66c7cbcbc5-2w8gh"]
Jul 06 20:54:41.763780 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:54:41.763746    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:54:41.778916 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:54:41.778893    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/586d1b5f-031b-4853-8204-297de26c8591-home\") pod \"e2e-lifecycle-model-1726c2fe-kserve-66c7cbcbc5-2w8gh\" (UID: \"586d1b5f-031b-4853-8204-297de26c8591\") " pod="ai-tenant-e2e-mt-1726c2fe/e2e-lifecycle-model-1726c2fe-kserve-66c7cbcbc5-2w8gh"
Jul 06 20:54:41.779015 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:54:41.778926    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/586d1b5f-031b-4853-8204-297de26c8591-dshm\") pod \"e2e-lifecycle-model-1726c2fe-kserve-66c7cbcbc5-2w8gh\" (UID: \"586d1b5f-031b-4853-8204-297de26c8591\") " pod="ai-tenant-e2e-mt-1726c2fe/e2e-lifecycle-model-1726c2fe-kserve-66c7cbcbc5-2w8gh"
Jul 06 20:54:41.779015 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:54:41.778966    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpglf\" (UniqueName: \"kubernetes.io/projected/586d1b5f-031b-4853-8204-297de26c8591-kube-api-access-tpglf\") pod \"e2e-lifecycle-model-1726c2fe-kserve-66c7cbcbc5-2w8gh\" (UID: \"586d1b5f-031b-4853-8204-297de26c8591\") " pod="ai-tenant-e2e-mt-1726c2fe/e2e-lifecycle-model-1726c2fe-kserve-66c7cbcbc5-2w8gh"
Jul 06 20:54:41.779015 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:54:41.778993    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/586d1b5f-031b-4853-8204-297de26c8591-model-cache\") pod \"e2e-lifecycle-model-1726c2fe-kserve-66c7cbcbc5-2w8gh\" (UID: \"586d1b5f-031b-4853-8204-297de26c8591\") " pod="ai-tenant-e2e-mt-1726c2fe/e2e-lifecycle-model-1726c2fe-kserve-66c7cbcbc5-2w8gh"
Jul 06 20:54:41.779128 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:54:41.779025    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/586d1b5f-031b-4853-8204-297de26c8591-tls-certs\") pod \"e2e-lifecycle-model-1726c2fe-kserve-66c7cbcbc5-2w8gh\" (UID: \"586d1b5f-031b-4853-8204-297de26c8591\") " pod="ai-tenant-e2e-mt-1726c2fe/e2e-lifecycle-model-1726c2fe-kserve-66c7cbcbc5-2w8gh"
Jul 06 20:54:41.779168 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:54:41.779126    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/586d1b5f-031b-4853-8204-297de26c8591-tmp-dir\") pod \"e2e-lifecycle-model-1726c2fe-kserve-66c7cbcbc5-2w8gh\" (UID: \"586d1b5f-031b-4853-8204-297de26c8591\") " pod="ai-tenant-e2e-mt-1726c2fe/e2e-lifecycle-model-1726c2fe-kserve-66c7cbcbc5-2w8gh"
Jul 06 20:54:41.879886 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:54:41.879852    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/586d1b5f-031b-4853-8204-297de26c8591-home\") pod \"e2e-lifecycle-model-1726c2fe-kserve-66c7cbcbc5-2w8gh\" (UID: \"586d1b5f-031b-4853-8204-297de26c8591\") " pod="ai-tenant-e2e-mt-1726c2fe/e2e-lifecycle-model-1726c2fe-kserve-66c7cbcbc5-2w8gh"
Jul 06 20:54:41.880052 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:54:41.879893    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/586d1b5f-031b-4853-8204-297de26c8591-dshm\") pod \"e2e-lifecycle-model-1726c2fe-kserve-66c7cbcbc5-2w8gh\" (UID: \"586d1b5f-031b-4853-8204-297de26c8591\") " pod="ai-tenant-e2e-mt-1726c2fe/e2e-lifecycle-model-1726c2fe-kserve-66c7cbcbc5-2w8gh"
Jul 06 20:54:41.880052 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:54:41.880030    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpglf\" (UniqueName: \"kubernetes.io/projected/586d1b5f-031b-4853-8204-297de26c8591-kube-api-access-tpglf\") pod \"e2e-lifecycle-model-1726c2fe-kserve-66c7cbcbc5-2w8gh\" (UID: \"586d1b5f-031b-4853-8204-297de26c8591\") " pod="ai-tenant-e2e-mt-1726c2fe/e2e-lifecycle-model-1726c2fe-kserve-66c7cbcbc5-2w8gh"
Jul 06 20:54:41.880137 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:54:41.880057    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/586d1b5f-031b-4853-8204-297de26c8591-model-cache\") pod \"e2e-lifecycle-model-1726c2fe-kserve-66c7cbcbc5-2w8gh\" (UID: \"586d1b5f-031b-4853-8204-297de26c8591\") " pod="ai-tenant-e2e-mt-1726c2fe/e2e-lifecycle-model-1726c2fe-kserve-66c7cbcbc5-2w8gh"
Jul 06 20:54:41.880137 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:54:41.880108    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/586d1b5f-031b-4853-8204-297de26c8591-tls-certs\") pod \"e2e-lifecycle-model-1726c2fe-kserve-66c7cbcbc5-2w8gh\" (UID: \"586d1b5f-031b-4853-8204-297de26c8591\") " pod="ai-tenant-e2e-mt-1726c2fe/e2e-lifecycle-model-1726c2fe-kserve-66c7cbcbc5-2w8gh"
Jul 06 20:54:41.880209 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:54:41.880163    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/586d1b5f-031b-4853-8204-297de26c8591-tmp-dir\") pod \"e2e-lifecycle-model-1726c2fe-kserve-66c7cbcbc5-2w8gh\" (UID: \"586d1b5f-031b-4853-8204-297de26c8591\") " pod="ai-tenant-e2e-mt-1726c2fe/e2e-lifecycle-model-1726c2fe-kserve-66c7cbcbc5-2w8gh"
Jul 06 20:54:41.880371 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:54:41.880348    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/586d1b5f-031b-4853-8204-297de26c8591-home\") pod \"e2e-lifecycle-model-1726c2fe-kserve-66c7cbcbc5-2w8gh\" (UID: \"586d1b5f-031b-4853-8204-297de26c8591\") " pod="ai-tenant-e2e-mt-1726c2fe/e2e-lifecycle-model-1726c2fe-kserve-66c7cbcbc5-2w8gh"
Jul 06 20:54:41.880564 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:54:41.880429    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/586d1b5f-031b-4853-8204-297de26c8591-model-cache\") pod \"e2e-lifecycle-model-1726c2fe-kserve-66c7cbcbc5-2w8gh\" (UID: \"586d1b5f-031b-4853-8204-297de26c8591\") " pod="ai-tenant-e2e-mt-1726c2fe/e2e-lifecycle-model-1726c2fe-kserve-66c7cbcbc5-2w8gh"
Jul 06 20:54:41.880564 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:54:41.880496    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/586d1b5f-031b-4853-8204-297de26c8591-tmp-dir\") pod \"e2e-lifecycle-model-1726c2fe-kserve-66c7cbcbc5-2w8gh\" (UID: \"586d1b5f-031b-4853-8204-297de26c8591\") " pod="ai-tenant-e2e-mt-1726c2fe/e2e-lifecycle-model-1726c2fe-kserve-66c7cbcbc5-2w8gh"
Jul 06 20:54:41.882171 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:54:41.882149    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/586d1b5f-031b-4853-8204-297de26c8591-dshm\") pod \"e2e-lifecycle-model-1726c2fe-kserve-66c7cbcbc5-2w8gh\" (UID: \"586d1b5f-031b-4853-8204-297de26c8591\") " pod="ai-tenant-e2e-mt-1726c2fe/e2e-lifecycle-model-1726c2fe-kserve-66c7cbcbc5-2w8gh"
Jul 06 20:54:41.882478 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:54:41.882462    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/586d1b5f-031b-4853-8204-297de26c8591-tls-certs\") pod \"e2e-lifecycle-model-1726c2fe-kserve-66c7cbcbc5-2w8gh\" (UID: \"586d1b5f-031b-4853-8204-297de26c8591\") " pod="ai-tenant-e2e-mt-1726c2fe/e2e-lifecycle-model-1726c2fe-kserve-66c7cbcbc5-2w8gh"
Jul 06 20:54:41.888613 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:54:41.888591    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpglf\" (UniqueName: \"kubernetes.io/projected/586d1b5f-031b-4853-8204-297de26c8591-kube-api-access-tpglf\") pod \"e2e-lifecycle-model-1726c2fe-kserve-66c7cbcbc5-2w8gh\" (UID: \"586d1b5f-031b-4853-8204-297de26c8591\") " pod="ai-tenant-e2e-mt-1726c2fe/e2e-lifecycle-model-1726c2fe-kserve-66c7cbcbc5-2w8gh"
Jul 06 20:54:42.042787 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:54:42.042678    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-1726c2fe/e2e-lifecycle-model-1726c2fe-kserve-66c7cbcbc5-2w8gh"
Jul 06 20:54:42.174401 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:54:42.174361    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-1726c2fe/e2e-lifecycle-model-1726c2fe-kserve-66c7cbcbc5-2w8gh"]
Jul 06 20:54:42.177647 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:54:42.177615    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod586d1b5f_031b_4853_8204_297de26c8591.slice/crio-7ee6a5de7ec22887a91f60d7cfd3873c0181537bb2012814cc08e859083b93e4 WatchSource:0}: Error finding container 7ee6a5de7ec22887a91f60d7cfd3873c0181537bb2012814cc08e859083b93e4: Status 404 returned error can't find the container with id 7ee6a5de7ec22887a91f60d7cfd3873c0181537bb2012814cc08e859083b93e4
Jul 06 20:54:42.327085 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:54:42.327056    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-1726c2fe/e2e-lifecycle-model-1726c2fe-kserve-66c7cbcbc5-2w8gh" event={"ID":"586d1b5f-031b-4853-8204-297de26c8591","Type":"ContainerStarted","Data":"3f253f7a7501b5ae9c76d7c56f7b5fa851ef33649c91624857ce8fb24c365816"}
Jul 06 20:54:42.327247 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:54:42.327093    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-1726c2fe/e2e-lifecycle-model-1726c2fe-kserve-66c7cbcbc5-2w8gh" event={"ID":"586d1b5f-031b-4853-8204-297de26c8591","Type":"ContainerStarted","Data":"7ee6a5de7ec22887a91f60d7cfd3873c0181537bb2012814cc08e859083b93e4"}
Jul 06 20:54:42.347452 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:54:42.347406    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-1726c2fe/e2e-lifecycle-model-1726c2fe-kserve-66c7cbcbc5-2w8gh" podStartSLOduration=1.347391231 podStartE2EDuration="1.347391231s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:54:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:54:42.345485987 +0000 UTC m=+1846.061596649" watchObservedRunningTime="2026-07-06 20:54:42.347391231 +0000 UTC m=+1846.063501890"
Jul 06 20:54:44.890690 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:54:44.890657    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-clwd5"]
Jul 06 20:54:52.042789 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:54:52.042758    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-1726c2fe/e2e-lifecycle-model-1726c2fe-kserve-66c7cbcbc5-2w8gh"
Jul 06 20:54:52.042789 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:54:52.042798    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-1726c2fe/e2e-lifecycle-model-1726c2fe-kserve-66c7cbcbc5-2w8gh"
Jul 06 20:54:52.055017 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:54:52.054984    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-1726c2fe/e2e-lifecycle-model-1726c2fe-kserve-66c7cbcbc5-2w8gh"
Jul 06 20:54:52.384129 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:54:52.384099    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-1726c2fe/e2e-lifecycle-model-1726c2fe-kserve-66c7cbcbc5-2w8gh"
Jul 06 20:55:08.018468 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:55:08.018435    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-clwd5"]
Jul 06 20:55:10.407013 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:55:10.406976    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:55:10.824188 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:55:10.824143    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:55:16.495401 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:55:16.495367    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:55:21.473912 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:55:21.473832    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-1726c2fe/e2e-lifecycle-model-1726c2fe-kserve-66c7cbcbc5-2w8gh"]
Jul 06 20:55:21.570489 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:55:21.570457    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:55:21.885606 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:55:21.885572    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-clwd5"]
Jul 06 20:55:36.494065 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:55:36.494012    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-mt-1726c2fe/e2e-lifecycle-model-1726c2fe-kserve-66c7cbcbc5-2w8gh" podUID="586d1b5f-031b-4853-8204-297de26c8591" containerName="main" containerID="cri-o://3f253f7a7501b5ae9c76d7c56f7b5fa851ef33649c91624857ce8fb24c365816" gracePeriod=45
Jul 06 20:55:36.741593 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:55:36.741569    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-1726c2fe/e2e-lifecycle-model-1726c2fe-kserve-66c7cbcbc5-2w8gh"
Jul 06 20:55:36.776960 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:55:36.776884    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tpglf\" (UniqueName: \"kubernetes.io/projected/586d1b5f-031b-4853-8204-297de26c8591-kube-api-access-tpglf\") pod \"586d1b5f-031b-4853-8204-297de26c8591\" (UID: \"586d1b5f-031b-4853-8204-297de26c8591\") "
Jul 06 20:55:36.776960 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:55:36.776957    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/586d1b5f-031b-4853-8204-297de26c8591-tmp-dir\") pod \"586d1b5f-031b-4853-8204-297de26c8591\" (UID: \"586d1b5f-031b-4853-8204-297de26c8591\") "
Jul 06 20:55:36.777172 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:55:36.777072    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/586d1b5f-031b-4853-8204-297de26c8591-home\") pod \"586d1b5f-031b-4853-8204-297de26c8591\" (UID: \"586d1b5f-031b-4853-8204-297de26c8591\") "
Jul 06 20:55:36.777172 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:55:36.777095    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/586d1b5f-031b-4853-8204-297de26c8591-dshm\") pod \"586d1b5f-031b-4853-8204-297de26c8591\" (UID: \"586d1b5f-031b-4853-8204-297de26c8591\") "
Jul 06 20:55:36.777172 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:55:36.777120    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/586d1b5f-031b-4853-8204-297de26c8591-tls-certs\") pod \"586d1b5f-031b-4853-8204-297de26c8591\" (UID: \"586d1b5f-031b-4853-8204-297de26c8591\") "
Jul 06 20:55:36.777172 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:55:36.777162    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/586d1b5f-031b-4853-8204-297de26c8591-model-cache\") pod \"586d1b5f-031b-4853-8204-297de26c8591\" (UID: \"586d1b5f-031b-4853-8204-297de26c8591\") "
Jul 06 20:55:36.777390 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:55:36.777225    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/586d1b5f-031b-4853-8204-297de26c8591-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "586d1b5f-031b-4853-8204-297de26c8591" (UID: "586d1b5f-031b-4853-8204-297de26c8591"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 20:55:36.777390 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:55:36.777344    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/586d1b5f-031b-4853-8204-297de26c8591-home" (OuterVolumeSpecName: "home") pod "586d1b5f-031b-4853-8204-297de26c8591" (UID: "586d1b5f-031b-4853-8204-297de26c8591"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 20:55:36.777500 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:55:36.777457    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/586d1b5f-031b-4853-8204-297de26c8591-model-cache" (OuterVolumeSpecName: "model-cache") pod "586d1b5f-031b-4853-8204-297de26c8591" (UID: "586d1b5f-031b-4853-8204-297de26c8591"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 20:55:36.777500 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:55:36.777488    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/586d1b5f-031b-4853-8204-297de26c8591-home\") on node \"ip-10-0-141-123.ec2.internal\" DevicePath \"\""
Jul 06 20:55:36.777613 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:55:36.777505    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/586d1b5f-031b-4853-8204-297de26c8591-tmp-dir\") on node \"ip-10-0-141-123.ec2.internal\" DevicePath \"\""
Jul 06 20:55:36.779492 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:55:36.779448    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/586d1b5f-031b-4853-8204-297de26c8591-dshm" (OuterVolumeSpecName: "dshm") pod "586d1b5f-031b-4853-8204-297de26c8591" (UID: "586d1b5f-031b-4853-8204-297de26c8591"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 20:55:36.779660 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:55:36.779634    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/586d1b5f-031b-4853-8204-297de26c8591-kube-api-access-tpglf" (OuterVolumeSpecName: "kube-api-access-tpglf") pod "586d1b5f-031b-4853-8204-297de26c8591" (UID: "586d1b5f-031b-4853-8204-297de26c8591"). InnerVolumeSpecName "kube-api-access-tpglf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 20:55:36.779747 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:55:36.779643    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/586d1b5f-031b-4853-8204-297de26c8591-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "586d1b5f-031b-4853-8204-297de26c8591" (UID: "586d1b5f-031b-4853-8204-297de26c8591"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 20:55:36.878178 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:55:36.878147    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/586d1b5f-031b-4853-8204-297de26c8591-dshm\") on node \"ip-10-0-141-123.ec2.internal\" DevicePath \"\""
Jul 06 20:55:36.878178 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:55:36.878168    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/586d1b5f-031b-4853-8204-297de26c8591-tls-certs\") on node \"ip-10-0-141-123.ec2.internal\" DevicePath \"\""
Jul 06 20:55:36.878178 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:55:36.878177    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/586d1b5f-031b-4853-8204-297de26c8591-model-cache\") on node \"ip-10-0-141-123.ec2.internal\" DevicePath \"\""
Jul 06 20:55:36.878465 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:55:36.878189    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpglf\" (UniqueName: \"kubernetes.io/projected/586d1b5f-031b-4853-8204-297de26c8591-kube-api-access-tpglf\") on node \"ip-10-0-141-123.ec2.internal\" DevicePath \"\""
Jul 06 20:55:37.587417 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:55:37.587378    2563 generic.go:356] "Generic (PLEG): container finished" podID="586d1b5f-031b-4853-8204-297de26c8591" containerID="3f253f7a7501b5ae9c76d7c56f7b5fa851ef33649c91624857ce8fb24c365816" exitCode=0
Jul 06 20:55:37.587873 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:55:37.587438    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-1726c2fe/e2e-lifecycle-model-1726c2fe-kserve-66c7cbcbc5-2w8gh"
Jul 06 20:55:37.587873 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:55:37.587461    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-1726c2fe/e2e-lifecycle-model-1726c2fe-kserve-66c7cbcbc5-2w8gh" event={"ID":"586d1b5f-031b-4853-8204-297de26c8591","Type":"ContainerDied","Data":"3f253f7a7501b5ae9c76d7c56f7b5fa851ef33649c91624857ce8fb24c365816"}
Jul 06 20:55:37.587873 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:55:37.587506    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-1726c2fe/e2e-lifecycle-model-1726c2fe-kserve-66c7cbcbc5-2w8gh" event={"ID":"586d1b5f-031b-4853-8204-297de26c8591","Type":"ContainerDied","Data":"7ee6a5de7ec22887a91f60d7cfd3873c0181537bb2012814cc08e859083b93e4"}
Jul 06 20:55:37.587873 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:55:37.587522    2563 scope.go:117] "RemoveContainer" containerID="3f253f7a7501b5ae9c76d7c56f7b5fa851ef33649c91624857ce8fb24c365816"
Jul 06 20:55:37.595791 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:55:37.595771    2563 scope.go:117] "RemoveContainer" containerID="3f253f7a7501b5ae9c76d7c56f7b5fa851ef33649c91624857ce8fb24c365816"
Jul 06 20:55:37.596032 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:55:37.596013    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f253f7a7501b5ae9c76d7c56f7b5fa851ef33649c91624857ce8fb24c365816\": container with ID starting with 3f253f7a7501b5ae9c76d7c56f7b5fa851ef33649c91624857ce8fb24c365816 not found: ID does not exist" containerID="3f253f7a7501b5ae9c76d7c56f7b5fa851ef33649c91624857ce8fb24c365816"
Jul 06 20:55:37.596074 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:55:37.596040    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f253f7a7501b5ae9c76d7c56f7b5fa851ef33649c91624857ce8fb24c365816"} err="failed to get container status \"3f253f7a7501b5ae9c76d7c56f7b5fa851ef33649c91624857ce8fb24c365816\": rpc error: code = NotFound desc = could not find container \"3f253f7a7501b5ae9c76d7c56f7b5fa851ef33649c91624857ce8fb24c365816\": container with ID starting with 3f253f7a7501b5ae9c76d7c56f7b5fa851ef33649c91624857ce8fb24c365816 not found: ID does not exist"
Jul 06 20:55:37.604581 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:55:37.604560    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-1726c2fe/e2e-lifecycle-model-1726c2fe-kserve-66c7cbcbc5-2w8gh"]
Jul 06 20:55:37.610788 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:55:37.610761    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-1726c2fe/e2e-lifecycle-model-1726c2fe-kserve-66c7cbcbc5-2w8gh"]
Jul 06 20:55:38.581568 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:55:38.581539    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:55:38.834167 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:55:38.834089    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="586d1b5f-031b-4853-8204-297de26c8591" path="/var/lib/kubelet/pods/586d1b5f-031b-4853-8204-297de26c8591/volumes"
Jul 06 20:55:39.315314 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:55:39.315287    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:55:41.785778 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:55:41.785741    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-clwd5"]
Jul 06 20:55:44.945033 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:55:44.945002    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:55:45.019069 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:55:45.019041    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:55:45.240715 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:55:45.240633    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:55:45.455646 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:55:45.455615    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:55:46.558768 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:55:46.558698    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:55:46.653322 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:55:46.653289    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:55:46.753630 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:55:46.753595    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:55:47.534388 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:55:47.534353    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:55:47.630314 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:55:47.630264    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:55:47.730907 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:55:47.730880    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:55:49.149287 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:55:49.149256    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:55:51.303798 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:55:51.303766    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-clwd5"]
Jul 06 20:55:56.446816 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:55:56.446783    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:55:57.242719 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:55:57.242634    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:55:57.386690 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:55:57.386655    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:55:58.027982 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:55:58.027952    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:56:02.505441 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:56:02.505402    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-clwd5"]
Jul 06 20:56:05.215663 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:56:05.215632    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:56:05.474713 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:56:05.474627    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:56:05.580874 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:56:05.580845    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:56:15.982031 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:56:15.981998    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:56:16.327046 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:56:16.327014    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:56:37.344991 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:56:37.344954    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:56:43.577066 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:56:43.577034    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:56:43.681338 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:56:43.681302    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:56:43.958051 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:56:43.957959    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:56:44.386011 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:56:44.385976    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:56:44.506850 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:56:44.506822    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:56:47.918786 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:56:47.918716    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:56:48.185704 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:56:48.185599    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:56:48.327482 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:56:48.327451    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:56:48.679248 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:56:48.679214    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:56:48.781830 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:56:48.781797    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:56:52.431491 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:56:52.431458    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:56:52.784224 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:56:52.784134    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:56:53.091816 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:56:53.091787    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:56:53.339512 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:56:53.339482    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:56:53.685649 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:56:53.685616    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-clwd5"]
Jul 06 20:56:55.995520 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:56:55.995493    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:56:56.153292 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:56:56.153257    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:56:56.356140 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:56:56.356113    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:56:56.501772 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:56:56.501748    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:56:56.798913 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:56:56.798833    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:56:56.898526 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:56:56.898489    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:57:00.564442 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:57:00.564413    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:57:04.897872 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:57:04.897841    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-clwd5"]
Jul 06 20:57:09.581751 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:57:09.581693    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:57:09.745167 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:57:09.745137    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:57:18.624570 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:57:18.624534    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:57:18.721714 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:57:18.721686    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:57:24.344911 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:57:24.344876    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-8e767e/test-model-8e767e-kserve-5ff5b4877c-tmgwh"]
Jul 06 20:57:24.345308 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:57:24.345259    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="586d1b5f-031b-4853-8204-297de26c8591" containerName="main"
Jul 06 20:57:24.345308 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:57:24.345269    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="586d1b5f-031b-4853-8204-297de26c8591" containerName="main"
Jul 06 20:57:24.345380 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:57:24.345328    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="586d1b5f-031b-4853-8204-297de26c8591" containerName="main"
Jul 06 20:57:24.348667 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:57:24.348648    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-8e767e/test-model-8e767e-kserve-5ff5b4877c-tmgwh"
Jul 06 20:57:24.351773 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:57:24.351754    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-8e767e\"/\"default-dockercfg-xmb7j\""
Jul 06 20:57:24.351916 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:57:24.351826    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-8e767e\"/\"openshift-service-ca.crt\""
Jul 06 20:57:24.351916 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:57:24.351834    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-8e767e\"/\"test-model-8e767e-kserve-self-signed-certs\""
Jul 06 20:57:24.351916 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:57:24.351891    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-8e767e\"/\"kube-root-ca.crt\""
Jul 06 20:57:24.360622 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:57:24.360583    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-8e767e/test-model-8e767e-kserve-5ff5b4877c-tmgwh"]
Jul 06 20:57:24.390996 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:57:24.390964    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:57:24.426046 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:57:24.426014    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5rm9\" (UniqueName: \"kubernetes.io/projected/5751a850-4933-4177-bfb5-993c3fe958f2-kube-api-access-z5rm9\") pod \"test-model-8e767e-kserve-5ff5b4877c-tmgwh\" (UID: \"5751a850-4933-4177-bfb5-993c3fe958f2\") " pod="ai-tenant-e2e-inf-a-8e767e/test-model-8e767e-kserve-5ff5b4877c-tmgwh"
Jul 06 20:57:24.426197 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:57:24.426054    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5751a850-4933-4177-bfb5-993c3fe958f2-home\") pod \"test-model-8e767e-kserve-5ff5b4877c-tmgwh\" (UID: \"5751a850-4933-4177-bfb5-993c3fe958f2\") " pod="ai-tenant-e2e-inf-a-8e767e/test-model-8e767e-kserve-5ff5b4877c-tmgwh"
Jul 06 20:57:24.426197 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:57:24.426091    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5751a850-4933-4177-bfb5-993c3fe958f2-tmp-dir\") pod \"test-model-8e767e-kserve-5ff5b4877c-tmgwh\" (UID: \"5751a850-4933-4177-bfb5-993c3fe958f2\") " pod="ai-tenant-e2e-inf-a-8e767e/test-model-8e767e-kserve-5ff5b4877c-tmgwh"
Jul 06 20:57:24.426197 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:57:24.426145    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5751a850-4933-4177-bfb5-993c3fe958f2-model-cache\") pod \"test-model-8e767e-kserve-5ff5b4877c-tmgwh\" (UID: \"5751a850-4933-4177-bfb5-993c3fe958f2\") " pod="ai-tenant-e2e-inf-a-8e767e/test-model-8e767e-kserve-5ff5b4877c-tmgwh"
Jul 06 20:57:24.426197 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:57:24.426190    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5751a850-4933-4177-bfb5-993c3fe958f2-dshm\") pod \"test-model-8e767e-kserve-5ff5b4877c-tmgwh\" (UID: \"5751a850-4933-4177-bfb5-993c3fe958f2\") " pod="ai-tenant-e2e-inf-a-8e767e/test-model-8e767e-kserve-5ff5b4877c-tmgwh"
Jul 06 20:57:24.426336 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:57:24.426208    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5751a850-4933-4177-bfb5-993c3fe958f2-tls-certs\") pod \"test-model-8e767e-kserve-5ff5b4877c-tmgwh\" (UID: \"5751a850-4933-4177-bfb5-993c3fe958f2\") " pod="ai-tenant-e2e-inf-a-8e767e/test-model-8e767e-kserve-5ff5b4877c-tmgwh"
Jul 06 20:57:24.527052 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:57:24.527011    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5751a850-4933-4177-bfb5-993c3fe958f2-tls-certs\") pod \"test-model-8e767e-kserve-5ff5b4877c-tmgwh\" (UID: \"5751a850-4933-4177-bfb5-993c3fe958f2\") " pod="ai-tenant-e2e-inf-a-8e767e/test-model-8e767e-kserve-5ff5b4877c-tmgwh"
Jul 06 20:57:24.527248 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:57:24.527076    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5rm9\" (UniqueName: \"kubernetes.io/projected/5751a850-4933-4177-bfb5-993c3fe958f2-kube-api-access-z5rm9\") pod \"test-model-8e767e-kserve-5ff5b4877c-tmgwh\" (UID: \"5751a850-4933-4177-bfb5-993c3fe958f2\") " pod="ai-tenant-e2e-inf-a-8e767e/test-model-8e767e-kserve-5ff5b4877c-tmgwh"
Jul 06 20:57:24.527248 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:57:24.527106    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5751a850-4933-4177-bfb5-993c3fe958f2-home\") pod \"test-model-8e767e-kserve-5ff5b4877c-tmgwh\" (UID: \"5751a850-4933-4177-bfb5-993c3fe958f2\") " pod="ai-tenant-e2e-inf-a-8e767e/test-model-8e767e-kserve-5ff5b4877c-tmgwh"
Jul 06 20:57:24.527248 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:57:24.527134    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5751a850-4933-4177-bfb5-993c3fe958f2-tmp-dir\") pod \"test-model-8e767e-kserve-5ff5b4877c-tmgwh\" (UID: \"5751a850-4933-4177-bfb5-993c3fe958f2\") " pod="ai-tenant-e2e-inf-a-8e767e/test-model-8e767e-kserve-5ff5b4877c-tmgwh"
Jul 06 20:57:24.527248 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:57:24.527164    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5751a850-4933-4177-bfb5-993c3fe958f2-model-cache\") pod \"test-model-8e767e-kserve-5ff5b4877c-tmgwh\" (UID: \"5751a850-4933-4177-bfb5-993c3fe958f2\") " pod="ai-tenant-e2e-inf-a-8e767e/test-model-8e767e-kserve-5ff5b4877c-tmgwh"
Jul 06 20:57:24.527248 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:57:24.527226    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5751a850-4933-4177-bfb5-993c3fe958f2-dshm\") pod \"test-model-8e767e-kserve-5ff5b4877c-tmgwh\" (UID: \"5751a850-4933-4177-bfb5-993c3fe958f2\") " pod="ai-tenant-e2e-inf-a-8e767e/test-model-8e767e-kserve-5ff5b4877c-tmgwh"
Jul 06 20:57:24.527670 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:57:24.527643    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5751a850-4933-4177-bfb5-993c3fe958f2-tmp-dir\") pod \"test-model-8e767e-kserve-5ff5b4877c-tmgwh\" (UID: \"5751a850-4933-4177-bfb5-993c3fe958f2\") " pod="ai-tenant-e2e-inf-a-8e767e/test-model-8e767e-kserve-5ff5b4877c-tmgwh"
Jul 06 20:57:24.527670 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:57:24.527659    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5751a850-4933-4177-bfb5-993c3fe958f2-home\") pod \"test-model-8e767e-kserve-5ff5b4877c-tmgwh\" (UID: \"5751a850-4933-4177-bfb5-993c3fe958f2\") " pod="ai-tenant-e2e-inf-a-8e767e/test-model-8e767e-kserve-5ff5b4877c-tmgwh"
Jul 06 20:57:24.527825 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:57:24.527683    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5751a850-4933-4177-bfb5-993c3fe958f2-model-cache\") pod \"test-model-8e767e-kserve-5ff5b4877c-tmgwh\" (UID: \"5751a850-4933-4177-bfb5-993c3fe958f2\") " pod="ai-tenant-e2e-inf-a-8e767e/test-model-8e767e-kserve-5ff5b4877c-tmgwh"
Jul 06 20:57:24.529408 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:57:24.529386    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5751a850-4933-4177-bfb5-993c3fe958f2-dshm\") pod \"test-model-8e767e-kserve-5ff5b4877c-tmgwh\" (UID: \"5751a850-4933-4177-bfb5-993c3fe958f2\") " pod="ai-tenant-e2e-inf-a-8e767e/test-model-8e767e-kserve-5ff5b4877c-tmgwh"
Jul 06 20:57:24.529577 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:57:24.529561    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5751a850-4933-4177-bfb5-993c3fe958f2-tls-certs\") pod \"test-model-8e767e-kserve-5ff5b4877c-tmgwh\" (UID: \"5751a850-4933-4177-bfb5-993c3fe958f2\") " pod="ai-tenant-e2e-inf-a-8e767e/test-model-8e767e-kserve-5ff5b4877c-tmgwh"
Jul 06 20:57:24.540889 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:57:24.540864    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5rm9\" (UniqueName: \"kubernetes.io/projected/5751a850-4933-4177-bfb5-993c3fe958f2-kube-api-access-z5rm9\") pod \"test-model-8e767e-kserve-5ff5b4877c-tmgwh\" (UID: \"5751a850-4933-4177-bfb5-993c3fe958f2\") " pod="ai-tenant-e2e-inf-a-8e767e/test-model-8e767e-kserve-5ff5b4877c-tmgwh"
Jul 06 20:57:24.658443 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:57:24.658348    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-8e767e/test-model-8e767e-kserve-5ff5b4877c-tmgwh"
Jul 06 20:57:24.808468 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:57:24.808443    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-8e767e/test-model-8e767e-kserve-5ff5b4877c-tmgwh"]
Jul 06 20:57:24.810788 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:57:24.810761    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5751a850_4933_4177_bfb5_993c3fe958f2.slice/crio-0f0ba34f44d8c91fac4d421081be01b7e3ac36c9749e2c0c401a3118ab0a56ea WatchSource:0}: Error finding container 0f0ba34f44d8c91fac4d421081be01b7e3ac36c9749e2c0c401a3118ab0a56ea: Status 404 returned error can't find the container with id 0f0ba34f44d8c91fac4d421081be01b7e3ac36c9749e2c0c401a3118ab0a56ea
Jul 06 20:57:25.083412 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:57:25.083379    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-8e767e/test-model-8e767e-kserve-5ff5b4877c-tmgwh" event={"ID":"5751a850-4933-4177-bfb5-993c3fe958f2","Type":"ContainerStarted","Data":"560f37f995932367d6a447cb272ad2d161520457b4273a7883c2888247e4307d"}
Jul 06 20:57:25.083619 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:57:25.083422    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-8e767e/test-model-8e767e-kserve-5ff5b4877c-tmgwh" event={"ID":"5751a850-4933-4177-bfb5-993c3fe958f2","Type":"ContainerStarted","Data":"0f0ba34f44d8c91fac4d421081be01b7e3ac36c9749e2c0c401a3118ab0a56ea"}
Jul 06 20:57:25.102965 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:57:25.102914    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-8e767e/test-model-8e767e-kserve-5ff5b4877c-tmgwh" podStartSLOduration=1.102898879 podStartE2EDuration="1.102898879s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:57:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:57:25.101000909 +0000 UTC m=+2008.817111572" watchObservedRunningTime="2026-07-06 20:57:25.102898879 +0000 UTC m=+2008.819009540"
Jul 06 20:57:27.498309 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:57:27.498278    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-clwd5"]
Jul 06 20:57:34.658483 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:57:34.658451    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-8e767e/test-model-8e767e-kserve-5ff5b4877c-tmgwh"
Jul 06 20:57:34.658902 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:57:34.658497    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-8e767e/test-model-8e767e-kserve-5ff5b4877c-tmgwh"
Jul 06 20:57:34.670936 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:57:34.670908    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-8e767e/test-model-8e767e-kserve-5ff5b4877c-tmgwh"
Jul 06 20:57:35.143707 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:57:35.143678    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-8e767e/test-model-8e767e-kserve-5ff5b4877c-tmgwh"
Jul 06 20:57:37.885956 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:57:37.885922    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-clwd5"]
Jul 06 20:57:43.953939 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:57:43.953908    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-60229a/test-model-60229a-kserve-5889d8747c-pv9td"]
Jul 06 20:57:43.957651 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:57:43.957631    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-60229a/test-model-60229a-kserve-5889d8747c-pv9td"
Jul 06 20:57:43.966935 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:57:43.966912    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-60229a\"/\"openshift-service-ca.crt\""
Jul 06 20:57:43.967058 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:57:43.966953    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-60229a\"/\"test-model-60229a-kserve-self-signed-certs\""
Jul 06 20:57:43.967058 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:57:43.966957    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-60229a\"/\"default-dockercfg-f2jf8\""
Jul 06 20:57:43.967058 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:57:43.966998    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-60229a\"/\"kube-root-ca.crt\""
Jul 06 20:57:43.978920 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:57:43.978900    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-60229a/test-model-60229a-kserve-5889d8747c-pv9td"]
Jul 06 20:57:43.987662 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:57:43.987642    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:57:44.106889 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:57:44.106848    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da6d0156-a1ff-4416-afa1-b4d3b006bb5f-tmp-dir\") pod \"test-model-60229a-kserve-5889d8747c-pv9td\" (UID: \"da6d0156-a1ff-4416-afa1-b4d3b006bb5f\") " pod="ai-tenant-e2e-inf-b-60229a/test-model-60229a-kserve-5889d8747c-pv9td"
Jul 06 20:57:44.106889 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:57:44.106896    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da6d0156-a1ff-4416-afa1-b4d3b006bb5f-dshm\") pod \"test-model-60229a-kserve-5889d8747c-pv9td\" (UID: \"da6d0156-a1ff-4416-afa1-b4d3b006bb5f\") " pod="ai-tenant-e2e-inf-b-60229a/test-model-60229a-kserve-5889d8747c-pv9td"
Jul 06 20:57:44.107122 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:57:44.106916    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wrl5\" (UniqueName: \"kubernetes.io/projected/da6d0156-a1ff-4416-afa1-b4d3b006bb5f-kube-api-access-2wrl5\") pod \"test-model-60229a-kserve-5889d8747c-pv9td\" (UID: \"da6d0156-a1ff-4416-afa1-b4d3b006bb5f\") " pod="ai-tenant-e2e-inf-b-60229a/test-model-60229a-kserve-5889d8747c-pv9td"
Jul 06 20:57:44.107122 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:57:44.106964    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da6d0156-a1ff-4416-afa1-b4d3b006bb5f-home\") pod \"test-model-60229a-kserve-5889d8747c-pv9td\" (UID: \"da6d0156-a1ff-4416-afa1-b4d3b006bb5f\") " pod="ai-tenant-e2e-inf-b-60229a/test-model-60229a-kserve-5889d8747c-pv9td"
Jul 06 20:57:44.107122 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:57:44.107015    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da6d0156-a1ff-4416-afa1-b4d3b006bb5f-model-cache\") pod \"test-model-60229a-kserve-5889d8747c-pv9td\" (UID: \"da6d0156-a1ff-4416-afa1-b4d3b006bb5f\") " pod="ai-tenant-e2e-inf-b-60229a/test-model-60229a-kserve-5889d8747c-pv9td"
Jul 06 20:57:44.107122 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:57:44.107038    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da6d0156-a1ff-4416-afa1-b4d3b006bb5f-tls-certs\") pod \"test-model-60229a-kserve-5889d8747c-pv9td\" (UID: \"da6d0156-a1ff-4416-afa1-b4d3b006bb5f\") " pod="ai-tenant-e2e-inf-b-60229a/test-model-60229a-kserve-5889d8747c-pv9td"
Jul 06 20:57:44.207717 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:57:44.207623    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da6d0156-a1ff-4416-afa1-b4d3b006bb5f-model-cache\") pod \"test-model-60229a-kserve-5889d8747c-pv9td\" (UID: \"da6d0156-a1ff-4416-afa1-b4d3b006bb5f\") " pod="ai-tenant-e2e-inf-b-60229a/test-model-60229a-kserve-5889d8747c-pv9td"
Jul 06 20:57:44.207717 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:57:44.207677    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da6d0156-a1ff-4416-afa1-b4d3b006bb5f-tls-certs\") pod \"test-model-60229a-kserve-5889d8747c-pv9td\" (UID: \"da6d0156-a1ff-4416-afa1-b4d3b006bb5f\") " pod="ai-tenant-e2e-inf-b-60229a/test-model-60229a-kserve-5889d8747c-pv9td"
Jul 06 20:57:44.207974 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:57:44.207766    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da6d0156-a1ff-4416-afa1-b4d3b006bb5f-tmp-dir\") pod \"test-model-60229a-kserve-5889d8747c-pv9td\" (UID: \"da6d0156-a1ff-4416-afa1-b4d3b006bb5f\") " pod="ai-tenant-e2e-inf-b-60229a/test-model-60229a-kserve-5889d8747c-pv9td"
Jul 06 20:57:44.207974 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:57:44.207806    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da6d0156-a1ff-4416-afa1-b4d3b006bb5f-dshm\") pod \"test-model-60229a-kserve-5889d8747c-pv9td\" (UID: \"da6d0156-a1ff-4416-afa1-b4d3b006bb5f\") " pod="ai-tenant-e2e-inf-b-60229a/test-model-60229a-kserve-5889d8747c-pv9td"
Jul 06 20:57:44.207974 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:57:44.207840    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wrl5\" (UniqueName: \"kubernetes.io/projected/da6d0156-a1ff-4416-afa1-b4d3b006bb5f-kube-api-access-2wrl5\") pod \"test-model-60229a-kserve-5889d8747c-pv9td\" (UID: \"da6d0156-a1ff-4416-afa1-b4d3b006bb5f\") " pod="ai-tenant-e2e-inf-b-60229a/test-model-60229a-kserve-5889d8747c-pv9td"
Jul 06 20:57:44.207974 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:57:44.207892    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da6d0156-a1ff-4416-afa1-b4d3b006bb5f-home\") pod \"test-model-60229a-kserve-5889d8747c-pv9td\" (UID: \"da6d0156-a1ff-4416-afa1-b4d3b006bb5f\") " pod="ai-tenant-e2e-inf-b-60229a/test-model-60229a-kserve-5889d8747c-pv9td"
Jul 06 20:57:44.208192 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:57:44.208115    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da6d0156-a1ff-4416-afa1-b4d3b006bb5f-model-cache\") pod \"test-model-60229a-kserve-5889d8747c-pv9td\" (UID: \"da6d0156-a1ff-4416-afa1-b4d3b006bb5f\") " pod="ai-tenant-e2e-inf-b-60229a/test-model-60229a-kserve-5889d8747c-pv9td"
Jul 06 20:57:44.208192 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:57:44.208129    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da6d0156-a1ff-4416-afa1-b4d3b006bb5f-tmp-dir\") pod \"test-model-60229a-kserve-5889d8747c-pv9td\" (UID: \"da6d0156-a1ff-4416-afa1-b4d3b006bb5f\") " pod="ai-tenant-e2e-inf-b-60229a/test-model-60229a-kserve-5889d8747c-pv9td"
Jul 06 20:57:44.208285 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:57:44.208250    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da6d0156-a1ff-4416-afa1-b4d3b006bb5f-home\") pod \"test-model-60229a-kserve-5889d8747c-pv9td\" (UID: \"da6d0156-a1ff-4416-afa1-b4d3b006bb5f\") " pod="ai-tenant-e2e-inf-b-60229a/test-model-60229a-kserve-5889d8747c-pv9td"
Jul 06 20:57:44.210057 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:57:44.210034    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da6d0156-a1ff-4416-afa1-b4d3b006bb5f-dshm\") pod \"test-model-60229a-kserve-5889d8747c-pv9td\" (UID: \"da6d0156-a1ff-4416-afa1-b4d3b006bb5f\") " pod="ai-tenant-e2e-inf-b-60229a/test-model-60229a-kserve-5889d8747c-pv9td"
Jul 06 20:57:44.210286 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:57:44.210269    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da6d0156-a1ff-4416-afa1-b4d3b006bb5f-tls-certs\") pod \"test-model-60229a-kserve-5889d8747c-pv9td\" (UID: \"da6d0156-a1ff-4416-afa1-b4d3b006bb5f\") " pod="ai-tenant-e2e-inf-b-60229a/test-model-60229a-kserve-5889d8747c-pv9td"
Jul 06 20:57:44.216416 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:57:44.216393    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wrl5\" (UniqueName: \"kubernetes.io/projected/da6d0156-a1ff-4416-afa1-b4d3b006bb5f-kube-api-access-2wrl5\") pod \"test-model-60229a-kserve-5889d8747c-pv9td\" (UID: \"da6d0156-a1ff-4416-afa1-b4d3b006bb5f\") " pod="ai-tenant-e2e-inf-b-60229a/test-model-60229a-kserve-5889d8747c-pv9td"
Jul 06 20:57:44.267323 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:57:44.267281    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-60229a/test-model-60229a-kserve-5889d8747c-pv9td"
Jul 06 20:57:44.401660 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:57:44.401634    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-60229a/test-model-60229a-kserve-5889d8747c-pv9td"]
Jul 06 20:57:44.404461 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:57:44.404426    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podda6d0156_a1ff_4416_afa1_b4d3b006bb5f.slice/crio-ffcc144f560c48b6b26546b101cb3e5db469b94110d1566e2899205acdda770d WatchSource:0}: Error finding container ffcc144f560c48b6b26546b101cb3e5db469b94110d1566e2899205acdda770d: Status 404 returned error can't find the container with id ffcc144f560c48b6b26546b101cb3e5db469b94110d1566e2899205acdda770d
Jul 06 20:57:45.181503 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:57:45.181467    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-60229a/test-model-60229a-kserve-5889d8747c-pv9td" event={"ID":"da6d0156-a1ff-4416-afa1-b4d3b006bb5f","Type":"ContainerStarted","Data":"51447c17ab065aacf6c536fd751104983bf273bb569151f34bb687295f0b173e"}
Jul 06 20:57:45.181503 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:57:45.181510    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-60229a/test-model-60229a-kserve-5889d8747c-pv9td" event={"ID":"da6d0156-a1ff-4416-afa1-b4d3b006bb5f","Type":"ContainerStarted","Data":"ffcc144f560c48b6b26546b101cb3e5db469b94110d1566e2899205acdda770d"}
Jul 06 20:57:45.198857 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:57:45.198803    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-60229a/test-model-60229a-kserve-5889d8747c-pv9td" podStartSLOduration=2.19878667 podStartE2EDuration="2.19878667s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:57:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:57:45.197663319 +0000 UTC m=+2028.913774002" watchObservedRunningTime="2026-07-06 20:57:45.19878667 +0000 UTC m=+2028.914897331"
Jul 06 20:57:48.696257 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:57:48.696221    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-clwd5"]
Jul 06 20:57:54.268234 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:57:54.268190    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-60229a/test-model-60229a-kserve-5889d8747c-pv9td"
Jul 06 20:57:54.268705 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:57:54.268264    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-60229a/test-model-60229a-kserve-5889d8747c-pv9td"
Jul 06 20:57:54.281076 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:57:54.281046    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-60229a/test-model-60229a-kserve-5889d8747c-pv9td"
Jul 06 20:57:55.239870 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:57:55.239837    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-60229a/test-model-60229a-kserve-5889d8747c-pv9td"
Jul 06 20:58:01.489082 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:58:01.489044    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-clwd5"]
Jul 06 20:58:04.559624 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:58:04.559588    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-8e767e/test-model-8e767e-kserve-5ff5b4877c-tmgwh"]
Jul 06 20:58:04.582888 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:58:04.582858    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:58:05.512756 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:58:05.512709    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-60229a/test-model-60229a-kserve-5889d8747c-pv9td"]
Jul 06 20:58:05.541684 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:58:05.541652    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:58:05.675402 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:58:05.675372    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:58:06.086338 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:58:06.086307    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:58:06.185996 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:58:06.185966    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:58:06.384181 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:58:06.384096    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:58:11.176398 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:58:11.176359    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:58:17.031724 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:58:17.031691    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-clwd5"]
Jul 06 20:58:19.580383 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:58:19.580286    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-8e767e/test-model-8e767e-kserve-5ff5b4877c-tmgwh" podUID="5751a850-4933-4177-bfb5-993c3fe958f2" containerName="main" containerID="cri-o://560f37f995932367d6a447cb272ad2d161520457b4273a7883c2888247e4307d" gracePeriod=45
Jul 06 20:58:19.828339 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:58:19.828317    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-8e767e/test-model-8e767e-kserve-5ff5b4877c-tmgwh"
Jul 06 20:58:19.944031 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:58:19.943950    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5751a850-4933-4177-bfb5-993c3fe958f2-tmp-dir\") pod \"5751a850-4933-4177-bfb5-993c3fe958f2\" (UID: \"5751a850-4933-4177-bfb5-993c3fe958f2\") "
Jul 06 20:58:19.944031 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:58:19.944003    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z5rm9\" (UniqueName: \"kubernetes.io/projected/5751a850-4933-4177-bfb5-993c3fe958f2-kube-api-access-z5rm9\") pod \"5751a850-4933-4177-bfb5-993c3fe958f2\" (UID: \"5751a850-4933-4177-bfb5-993c3fe958f2\") "
Jul 06 20:58:19.944031 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:58:19.944028    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5751a850-4933-4177-bfb5-993c3fe958f2-home\") pod \"5751a850-4933-4177-bfb5-993c3fe958f2\" (UID: \"5751a850-4933-4177-bfb5-993c3fe958f2\") "
Jul 06 20:58:19.944314 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:58:19.944043    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5751a850-4933-4177-bfb5-993c3fe958f2-tls-certs\") pod \"5751a850-4933-4177-bfb5-993c3fe958f2\" (UID: \"5751a850-4933-4177-bfb5-993c3fe958f2\") "
Jul 06 20:58:19.944314 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:58:19.944181    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5751a850-4933-4177-bfb5-993c3fe958f2-model-cache\") pod \"5751a850-4933-4177-bfb5-993c3fe958f2\" (UID: \"5751a850-4933-4177-bfb5-993c3fe958f2\") "
Jul 06 20:58:19.944314 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:58:19.944249    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5751a850-4933-4177-bfb5-993c3fe958f2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5751a850-4933-4177-bfb5-993c3fe958f2" (UID: "5751a850-4933-4177-bfb5-993c3fe958f2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 20:58:19.944314 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:58:19.944296    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5751a850-4933-4177-bfb5-993c3fe958f2-home" (OuterVolumeSpecName: "home") pod "5751a850-4933-4177-bfb5-993c3fe958f2" (UID: "5751a850-4933-4177-bfb5-993c3fe958f2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 20:58:19.944492 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:58:19.944315    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5751a850-4933-4177-bfb5-993c3fe958f2-dshm\") pod \"5751a850-4933-4177-bfb5-993c3fe958f2\" (UID: \"5751a850-4933-4177-bfb5-993c3fe958f2\") "
Jul 06 20:58:19.944492 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:58:19.944422    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5751a850-4933-4177-bfb5-993c3fe958f2-model-cache" (OuterVolumeSpecName: "model-cache") pod "5751a850-4933-4177-bfb5-993c3fe958f2" (UID: "5751a850-4933-4177-bfb5-993c3fe958f2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 20:58:19.944625 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:58:19.944603    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5751a850-4933-4177-bfb5-993c3fe958f2-tmp-dir\") on node \"ip-10-0-141-123.ec2.internal\" DevicePath \"\""
Jul 06 20:58:19.944625 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:58:19.944624    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5751a850-4933-4177-bfb5-993c3fe958f2-home\") on node \"ip-10-0-141-123.ec2.internal\" DevicePath \"\""
Jul 06 20:58:19.944819 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:58:19.944638    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5751a850-4933-4177-bfb5-993c3fe958f2-model-cache\") on node \"ip-10-0-141-123.ec2.internal\" DevicePath \"\""
Jul 06 20:58:19.946239 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:58:19.946213    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5751a850-4933-4177-bfb5-993c3fe958f2-dshm" (OuterVolumeSpecName: "dshm") pod "5751a850-4933-4177-bfb5-993c3fe958f2" (UID: "5751a850-4933-4177-bfb5-993c3fe958f2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 20:58:19.946334 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:58:19.946268    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5751a850-4933-4177-bfb5-993c3fe958f2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5751a850-4933-4177-bfb5-993c3fe958f2" (UID: "5751a850-4933-4177-bfb5-993c3fe958f2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 20:58:19.946334 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:58:19.946289    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5751a850-4933-4177-bfb5-993c3fe958f2-kube-api-access-z5rm9" (OuterVolumeSpecName: "kube-api-access-z5rm9") pod "5751a850-4933-4177-bfb5-993c3fe958f2" (UID: "5751a850-4933-4177-bfb5-993c3fe958f2"). InnerVolumeSpecName "kube-api-access-z5rm9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 20:58:20.046033 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:58:20.045997    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5rm9\" (UniqueName: \"kubernetes.io/projected/5751a850-4933-4177-bfb5-993c3fe958f2-kube-api-access-z5rm9\") on node \"ip-10-0-141-123.ec2.internal\" DevicePath \"\""
Jul 06 20:58:20.046033 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:58:20.046024    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5751a850-4933-4177-bfb5-993c3fe958f2-tls-certs\") on node \"ip-10-0-141-123.ec2.internal\" DevicePath \"\""
Jul 06 20:58:20.046033 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:58:20.046034    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5751a850-4933-4177-bfb5-993c3fe958f2-dshm\") on node \"ip-10-0-141-123.ec2.internal\" DevicePath \"\""
Jul 06 20:58:20.348918 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:58:20.348887    2563 generic.go:356] "Generic (PLEG): container finished" podID="5751a850-4933-4177-bfb5-993c3fe958f2" containerID="560f37f995932367d6a447cb272ad2d161520457b4273a7883c2888247e4307d" exitCode=0
Jul 06 20:58:20.349085 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:58:20.348933    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-8e767e/test-model-8e767e-kserve-5ff5b4877c-tmgwh"
Jul 06 20:58:20.349085 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:58:20.348929    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-8e767e/test-model-8e767e-kserve-5ff5b4877c-tmgwh" event={"ID":"5751a850-4933-4177-bfb5-993c3fe958f2","Type":"ContainerDied","Data":"560f37f995932367d6a447cb272ad2d161520457b4273a7883c2888247e4307d"}
Jul 06 20:58:20.349085 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:58:20.349051    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-8e767e/test-model-8e767e-kserve-5ff5b4877c-tmgwh" event={"ID":"5751a850-4933-4177-bfb5-993c3fe958f2","Type":"ContainerDied","Data":"0f0ba34f44d8c91fac4d421081be01b7e3ac36c9749e2c0c401a3118ab0a56ea"}
Jul 06 20:58:20.349085 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:58:20.349078    2563 scope.go:117] "RemoveContainer" containerID="560f37f995932367d6a447cb272ad2d161520457b4273a7883c2888247e4307d"
Jul 06 20:58:20.358527 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:58:20.358510    2563 scope.go:117] "RemoveContainer" containerID="560f37f995932367d6a447cb272ad2d161520457b4273a7883c2888247e4307d"
Jul 06 20:58:20.358782 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:58:20.358761    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"560f37f995932367d6a447cb272ad2d161520457b4273a7883c2888247e4307d\": container with ID starting with 560f37f995932367d6a447cb272ad2d161520457b4273a7883c2888247e4307d not found: ID does not exist" containerID="560f37f995932367d6a447cb272ad2d161520457b4273a7883c2888247e4307d"
Jul 06 20:58:20.358839 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:58:20.358790    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"560f37f995932367d6a447cb272ad2d161520457b4273a7883c2888247e4307d"} err="failed to get container status \"560f37f995932367d6a447cb272ad2d161520457b4273a7883c2888247e4307d\": rpc error: code = NotFound desc = could not find container \"560f37f995932367d6a447cb272ad2d161520457b4273a7883c2888247e4307d\": container with ID starting with 560f37f995932367d6a447cb272ad2d161520457b4273a7883c2888247e4307d not found: ID does not exist"
Jul 06 20:58:20.372676 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:58:20.372650    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-8e767e/test-model-8e767e-kserve-5ff5b4877c-tmgwh"]
Jul 06 20:58:20.382039 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:58:20.382017    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-8e767e/test-model-8e767e-kserve-5ff5b4877c-tmgwh"]
Jul 06 20:58:20.532789 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:58:20.532754    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-60229a/test-model-60229a-kserve-5889d8747c-pv9td" podUID="da6d0156-a1ff-4416-afa1-b4d3b006bb5f" containerName="main" containerID="cri-o://51447c17ab065aacf6c536fd751104983bf273bb569151f34bb687295f0b173e" gracePeriod=45
Jul 06 20:58:20.788695 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:58:20.788673    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-60229a/test-model-60229a-kserve-5889d8747c-pv9td"
Jul 06 20:58:20.833394 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:58:20.833365    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5751a850-4933-4177-bfb5-993c3fe958f2" path="/var/lib/kubelet/pods/5751a850-4933-4177-bfb5-993c3fe958f2/volumes"
Jul 06 20:58:20.954830 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:58:20.954720    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da6d0156-a1ff-4416-afa1-b4d3b006bb5f-tmp-dir\") pod \"da6d0156-a1ff-4416-afa1-b4d3b006bb5f\" (UID: \"da6d0156-a1ff-4416-afa1-b4d3b006bb5f\") "
Jul 06 20:58:20.954830 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:58:20.954773    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da6d0156-a1ff-4416-afa1-b4d3b006bb5f-dshm\") pod \"da6d0156-a1ff-4416-afa1-b4d3b006bb5f\" (UID: \"da6d0156-a1ff-4416-afa1-b4d3b006bb5f\") "
Jul 06 20:58:20.954830 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:58:20.954811    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da6d0156-a1ff-4416-afa1-b4d3b006bb5f-home\") pod \"da6d0156-a1ff-4416-afa1-b4d3b006bb5f\" (UID: \"da6d0156-a1ff-4416-afa1-b4d3b006bb5f\") "
Jul 06 20:58:20.955116 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:58:20.954861    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2wrl5\" (UniqueName: \"kubernetes.io/projected/da6d0156-a1ff-4416-afa1-b4d3b006bb5f-kube-api-access-2wrl5\") pod \"da6d0156-a1ff-4416-afa1-b4d3b006bb5f\" (UID: \"da6d0156-a1ff-4416-afa1-b4d3b006bb5f\") "
Jul 06 20:58:20.955116 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:58:20.954891    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da6d0156-a1ff-4416-afa1-b4d3b006bb5f-model-cache\") pod \"da6d0156-a1ff-4416-afa1-b4d3b006bb5f\" (UID: \"da6d0156-a1ff-4416-afa1-b4d3b006bb5f\") "
Jul 06 20:58:20.955116 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:58:20.954919    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da6d0156-a1ff-4416-afa1-b4d3b006bb5f-tls-certs\") pod \"da6d0156-a1ff-4416-afa1-b4d3b006bb5f\" (UID: \"da6d0156-a1ff-4416-afa1-b4d3b006bb5f\") "
Jul 06 20:58:20.955116 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:58:20.955015    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da6d0156-a1ff-4416-afa1-b4d3b006bb5f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "da6d0156-a1ff-4416-afa1-b4d3b006bb5f" (UID: "da6d0156-a1ff-4416-afa1-b4d3b006bb5f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 20:58:20.955321 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:58:20.955114    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da6d0156-a1ff-4416-afa1-b4d3b006bb5f-home" (OuterVolumeSpecName: "home") pod "da6d0156-a1ff-4416-afa1-b4d3b006bb5f" (UID: "da6d0156-a1ff-4416-afa1-b4d3b006bb5f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 20:58:20.955321 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:58:20.955158    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da6d0156-a1ff-4416-afa1-b4d3b006bb5f-model-cache" (OuterVolumeSpecName: "model-cache") pod "da6d0156-a1ff-4416-afa1-b4d3b006bb5f" (UID: "da6d0156-a1ff-4416-afa1-b4d3b006bb5f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 20:58:20.955321 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:58:20.955292    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da6d0156-a1ff-4416-afa1-b4d3b006bb5f-model-cache\") on node \"ip-10-0-141-123.ec2.internal\" DevicePath \"\""
Jul 06 20:58:20.955321 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:58:20.955309    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da6d0156-a1ff-4416-afa1-b4d3b006bb5f-tmp-dir\") on node \"ip-10-0-141-123.ec2.internal\" DevicePath \"\""
Jul 06 20:58:20.955321 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:58:20.955320    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da6d0156-a1ff-4416-afa1-b4d3b006bb5f-home\") on node \"ip-10-0-141-123.ec2.internal\" DevicePath \"\""
Jul 06 20:58:20.957076 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:58:20.957050    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da6d0156-a1ff-4416-afa1-b4d3b006bb5f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "da6d0156-a1ff-4416-afa1-b4d3b006bb5f" (UID: "da6d0156-a1ff-4416-afa1-b4d3b006bb5f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 20:58:20.957462 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:58:20.957440    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da6d0156-a1ff-4416-afa1-b4d3b006bb5f-dshm" (OuterVolumeSpecName: "dshm") pod "da6d0156-a1ff-4416-afa1-b4d3b006bb5f" (UID: "da6d0156-a1ff-4416-afa1-b4d3b006bb5f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 20:58:20.957462 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:58:20.957455    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da6d0156-a1ff-4416-afa1-b4d3b006bb5f-kube-api-access-2wrl5" (OuterVolumeSpecName: "kube-api-access-2wrl5") pod "da6d0156-a1ff-4416-afa1-b4d3b006bb5f" (UID: "da6d0156-a1ff-4416-afa1-b4d3b006bb5f"). InnerVolumeSpecName "kube-api-access-2wrl5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 20:58:21.056787 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:58:21.056754    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wrl5\" (UniqueName: \"kubernetes.io/projected/da6d0156-a1ff-4416-afa1-b4d3b006bb5f-kube-api-access-2wrl5\") on node \"ip-10-0-141-123.ec2.internal\" DevicePath \"\""
Jul 06 20:58:21.056787 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:58:21.056783    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da6d0156-a1ff-4416-afa1-b4d3b006bb5f-tls-certs\") on node \"ip-10-0-141-123.ec2.internal\" DevicePath \"\""
Jul 06 20:58:21.056787 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:58:21.056792    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da6d0156-a1ff-4416-afa1-b4d3b006bb5f-dshm\") on node \"ip-10-0-141-123.ec2.internal\" DevicePath \"\""
Jul 06 20:58:21.355987 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:58:21.355957    2563 generic.go:356] "Generic (PLEG): container finished" podID="da6d0156-a1ff-4416-afa1-b4d3b006bb5f" containerID="51447c17ab065aacf6c536fd751104983bf273bb569151f34bb687295f0b173e" exitCode=0
Jul 06 20:58:21.356169 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:58:21.356001    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-60229a/test-model-60229a-kserve-5889d8747c-pv9td"
Jul 06 20:58:21.356169 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:58:21.356022    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-60229a/test-model-60229a-kserve-5889d8747c-pv9td" event={"ID":"da6d0156-a1ff-4416-afa1-b4d3b006bb5f","Type":"ContainerDied","Data":"51447c17ab065aacf6c536fd751104983bf273bb569151f34bb687295f0b173e"}
Jul 06 20:58:21.356169 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:58:21.356050    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-60229a/test-model-60229a-kserve-5889d8747c-pv9td" event={"ID":"da6d0156-a1ff-4416-afa1-b4d3b006bb5f","Type":"ContainerDied","Data":"ffcc144f560c48b6b26546b101cb3e5db469b94110d1566e2899205acdda770d"}
Jul 06 20:58:21.356169 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:58:21.356067    2563 scope.go:117] "RemoveContainer" containerID="51447c17ab065aacf6c536fd751104983bf273bb569151f34bb687295f0b173e"
Jul 06 20:58:21.372198 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:58:21.372178    2563 scope.go:117] "RemoveContainer" containerID="51447c17ab065aacf6c536fd751104983bf273bb569151f34bb687295f0b173e"
Jul 06 20:58:21.372496 ip-10-0-141-123 kubenswrapper[2563]: E0706 20:58:21.372477    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51447c17ab065aacf6c536fd751104983bf273bb569151f34bb687295f0b173e\": container with ID starting with 51447c17ab065aacf6c536fd751104983bf273bb569151f34bb687295f0b173e not found: ID does not exist" containerID="51447c17ab065aacf6c536fd751104983bf273bb569151f34bb687295f0b173e"
Jul 06 20:58:21.372562 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:58:21.372506    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51447c17ab065aacf6c536fd751104983bf273bb569151f34bb687295f0b173e"} err="failed to get container status \"51447c17ab065aacf6c536fd751104983bf273bb569151f34bb687295f0b173e\": rpc error: code = NotFound desc = could not find container \"51447c17ab065aacf6c536fd751104983bf273bb569151f34bb687295f0b173e\": container with ID starting with 51447c17ab065aacf6c536fd751104983bf273bb569151f34bb687295f0b173e not found: ID does not exist"
Jul 06 20:58:21.383078 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:58:21.383054    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-60229a/test-model-60229a-kserve-5889d8747c-pv9td"]
Jul 06 20:58:21.390629 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:58:21.390611    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-60229a/test-model-60229a-kserve-5889d8747c-pv9td"]
Jul 06 20:58:22.833660 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:58:22.833627    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da6d0156-a1ff-4416-afa1-b4d3b006bb5f" path="/var/lib/kubelet/pods/da6d0156-a1ff-4416-afa1-b4d3b006bb5f/volumes"
Jul 06 20:58:31.593509 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:58:31.593471    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:58:48.219440 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:58:48.219370    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:58:48.590779 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:58:48.590704    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:58:48.805888 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:58:48.805855    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:58:49.001933 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:58:49.001847    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:58:49.490921 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:58:49.490887    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:58:49.628742 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:58:49.628691    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:58:49.725505 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:58:49.725473    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:58:49.899925 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:58:49.899892    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:58:52.389115 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:58:52.389079    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-clwd5"]
Jul 06 20:58:55.459555 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:58:55.459522    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:58:55.527541 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:58:55.527514    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:58:55.744767 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:58:55.744648    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:58:55.988798 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:58:55.988763    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:58:56.444179 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:58:56.444145    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:58:56.837189 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:58:56.837165    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-2m7gh_bb2a41aa-a9fc-4ec5-9b78-5e22c1fd301b/console-operator/1.log"
Jul 06 20:58:56.840536 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:58:56.840517    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-2m7gh_bb2a41aa-a9fc-4ec5-9b78-5e22c1fd301b/console-operator/1.log"
Jul 06 20:58:56.850236 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:58:56.850213    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:58:57.352005 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:58:57.351965    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:58:59.879548 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:58:59.879519    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:59:01.135983 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:59:01.135923    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:59:04.267142 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:59:04.267111    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:59:04.732985 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:59:04.732949    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:59:05.053776 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:59:05.053751    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:59:05.497671 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:59:05.497580    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:59:05.498482 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:59:05.498453    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-clwd5"]
Jul 06 20:59:07.967993 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:59:07.967963    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:59:08.072616 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:59:08.072564    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:59:08.187357 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:59:08.187327    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:59:08.350702 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:59:08.350666    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:59:08.772232 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:59:08.772149    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:59:08.872531 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:59:08.872500    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:59:11.182829 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:59:11.182800    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:59:11.285208 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:59:11.285177    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:59:12.385052 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:59:12.385014    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:59:15.302009 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:59:15.301970    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-clwd5"]
Jul 06 20:59:20.735996 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:59:20.735962    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:59:21.148837 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:59:21.148800    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:59:21.277649 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:59:21.277619    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:59:21.378583 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:59:21.378542    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:59:21.697170 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:59:21.697128    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:59:26.708030 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:59:26.707995    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-clwd5"]
Jul 06 20:59:27.526152 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:59:27.526120    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:59:27.629198 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:59:27.629162    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:59:27.852960 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:59:27.852928    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:59:28.074678 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:59:28.074648    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:59:28.261158 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:59:28.261060    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:59:28.652553 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:59:28.652517    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:59:28.822932 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:59:28.822902    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:59:28.921014 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:59:28.920936    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:59:29.240359 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:59:29.240265    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:59:34.853437 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:59:34.853408    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:59:34.931266 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:59:34.931224    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:59:35.136405 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:59:35.136323    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:59:35.370925 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:59:35.370891    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:59:35.721089 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:59:35.721061    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:59:36.070649 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:59:36.070614    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:59:36.340847 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:59:36.340766    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:59:38.498339 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:59:38.498305    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-clwd5"]
Jul 06 20:59:38.939613 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:59:38.939578    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:59:39.095832 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:59:39.095799    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:59:39.338924 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:59:39.338896    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:59:39.692670 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:59:39.692585    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:59:40.076043 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:59:40.076013    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:59:40.174354 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:59:40.174322    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:59:43.460710 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:59:43.460672    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:59:43.831935 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:59:43.831901    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:59:44.505945 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:59:44.505906    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:59:44.839802 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:59:44.839772    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:59:47.148754 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:59:47.148712    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:59:47.502055 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:59:47.501979    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:59:47.668349 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:59:47.668322    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:59:48.476909 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:59:48.476830    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:59:48.698043 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:59:48.698010    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:59:50.204058 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:59:50.204019    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-clwd5"]
Jul 06 20:59:51.633900 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:59:51.633849    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:59:55.062886 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:59:55.060059    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-544d30/auth-test-model-544d30-kserve-84f7bfc47f-2xgt6"]
Jul 06 20:59:55.062886 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:59:55.060905    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da6d0156-a1ff-4416-afa1-b4d3b006bb5f" containerName="main"
Jul 06 20:59:55.062886 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:59:55.060927    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="da6d0156-a1ff-4416-afa1-b4d3b006bb5f" containerName="main"
Jul 06 20:59:55.062886 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:59:55.060983    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5751a850-4933-4177-bfb5-993c3fe958f2" containerName="main"
Jul 06 20:59:55.062886 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:59:55.060991    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5751a850-4933-4177-bfb5-993c3fe958f2" containerName="main"
Jul 06 20:59:55.062886 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:59:55.061168    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="5751a850-4933-4177-bfb5-993c3fe958f2" containerName="main"
Jul 06 20:59:55.062886 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:59:55.061191    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="da6d0156-a1ff-4416-afa1-b4d3b006bb5f" containerName="main"
Jul 06 20:59:55.065219 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:59:55.065094    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-544d30/auth-test-model-544d30-kserve-84f7bfc47f-2xgt6"
Jul 06 20:59:55.068018 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:59:55.067998    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-544d30\"/\"default-dockercfg-pjfp2\""
Jul 06 20:59:55.068799 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:59:55.068780    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-544d30\"/\"kube-root-ca.crt\""
Jul 06 20:59:55.068952 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:59:55.068781    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-544d30\"/\"auth-test-model-544d30-kserve-self-signed-certs\""
Jul 06 20:59:55.068952 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:59:55.068811    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-544d30\"/\"openshift-service-ca.crt\""
Jul 06 20:59:55.072783 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:59:55.072761    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-544d30/auth-test-model-544d30-kserve-84f7bfc47f-2xgt6"]
Jul 06 20:59:55.099541 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:59:55.099510    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 20:59:55.196583 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:59:55.196539    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb551609-543a-47a6-aad7-515f3001dd22-tmp-dir\") pod \"auth-test-model-544d30-kserve-84f7bfc47f-2xgt6\" (UID: \"eb551609-543a-47a6-aad7-515f3001dd22\") " pod="ai-tenant-e2e-shared-a-544d30/auth-test-model-544d30-kserve-84f7bfc47f-2xgt6"
Jul 06 20:59:55.196825 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:59:55.196645    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb551609-543a-47a6-aad7-515f3001dd22-home\") pod \"auth-test-model-544d30-kserve-84f7bfc47f-2xgt6\" (UID: \"eb551609-543a-47a6-aad7-515f3001dd22\") " pod="ai-tenant-e2e-shared-a-544d30/auth-test-model-544d30-kserve-84f7bfc47f-2xgt6"
Jul 06 20:59:55.196825 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:59:55.196705    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s25j5\" (UniqueName: \"kubernetes.io/projected/eb551609-543a-47a6-aad7-515f3001dd22-kube-api-access-s25j5\") pod \"auth-test-model-544d30-kserve-84f7bfc47f-2xgt6\" (UID: \"eb551609-543a-47a6-aad7-515f3001dd22\") " pod="ai-tenant-e2e-shared-a-544d30/auth-test-model-544d30-kserve-84f7bfc47f-2xgt6"
Jul 06 20:59:55.196825 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:59:55.196759    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb551609-543a-47a6-aad7-515f3001dd22-tls-certs\") pod \"auth-test-model-544d30-kserve-84f7bfc47f-2xgt6\" (UID: \"eb551609-543a-47a6-aad7-515f3001dd22\") " pod="ai-tenant-e2e-shared-a-544d30/auth-test-model-544d30-kserve-84f7bfc47f-2xgt6"
Jul 06 20:59:55.196825 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:59:55.196792    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb551609-543a-47a6-aad7-515f3001dd22-dshm\") pod \"auth-test-model-544d30-kserve-84f7bfc47f-2xgt6\" (UID: \"eb551609-543a-47a6-aad7-515f3001dd22\") " pod="ai-tenant-e2e-shared-a-544d30/auth-test-model-544d30-kserve-84f7bfc47f-2xgt6"
Jul 06 20:59:55.197021 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:59:55.196879    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb551609-543a-47a6-aad7-515f3001dd22-model-cache\") pod \"auth-test-model-544d30-kserve-84f7bfc47f-2xgt6\" (UID: \"eb551609-543a-47a6-aad7-515f3001dd22\") " pod="ai-tenant-e2e-shared-a-544d30/auth-test-model-544d30-kserve-84f7bfc47f-2xgt6"
Jul 06 20:59:55.298091 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:59:55.298059    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s25j5\" (UniqueName: \"kubernetes.io/projected/eb551609-543a-47a6-aad7-515f3001dd22-kube-api-access-s25j5\") pod \"auth-test-model-544d30-kserve-84f7bfc47f-2xgt6\" (UID: \"eb551609-543a-47a6-aad7-515f3001dd22\") " pod="ai-tenant-e2e-shared-a-544d30/auth-test-model-544d30-kserve-84f7bfc47f-2xgt6"
Jul 06 20:59:55.298299 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:59:55.298095    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb551609-543a-47a6-aad7-515f3001dd22-tls-certs\") pod \"auth-test-model-544d30-kserve-84f7bfc47f-2xgt6\" (UID: \"eb551609-543a-47a6-aad7-515f3001dd22\") " pod="ai-tenant-e2e-shared-a-544d30/auth-test-model-544d30-kserve-84f7bfc47f-2xgt6"
Jul 06 20:59:55.298299 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:59:55.298121    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb551609-543a-47a6-aad7-515f3001dd22-dshm\") pod \"auth-test-model-544d30-kserve-84f7bfc47f-2xgt6\" (UID: \"eb551609-543a-47a6-aad7-515f3001dd22\") " pod="ai-tenant-e2e-shared-a-544d30/auth-test-model-544d30-kserve-84f7bfc47f-2xgt6"
Jul 06 20:59:55.298299 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:59:55.298153    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb551609-543a-47a6-aad7-515f3001dd22-model-cache\") pod \"auth-test-model-544d30-kserve-84f7bfc47f-2xgt6\" (UID: \"eb551609-543a-47a6-aad7-515f3001dd22\") " pod="ai-tenant-e2e-shared-a-544d30/auth-test-model-544d30-kserve-84f7bfc47f-2xgt6"
Jul 06 20:59:55.298299 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:59:55.298178    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb551609-543a-47a6-aad7-515f3001dd22-tmp-dir\") pod \"auth-test-model-544d30-kserve-84f7bfc47f-2xgt6\" (UID: \"eb551609-543a-47a6-aad7-515f3001dd22\") " pod="ai-tenant-e2e-shared-a-544d30/auth-test-model-544d30-kserve-84f7bfc47f-2xgt6"
Jul 06 20:59:55.298299 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:59:55.298218    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb551609-543a-47a6-aad7-515f3001dd22-home\") pod \"auth-test-model-544d30-kserve-84f7bfc47f-2xgt6\" (UID: \"eb551609-543a-47a6-aad7-515f3001dd22\") " pod="ai-tenant-e2e-shared-a-544d30/auth-test-model-544d30-kserve-84f7bfc47f-2xgt6"
Jul 06 20:59:55.298592 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:59:55.298564    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb551609-543a-47a6-aad7-515f3001dd22-tmp-dir\") pod \"auth-test-model-544d30-kserve-84f7bfc47f-2xgt6\" (UID: \"eb551609-543a-47a6-aad7-515f3001dd22\") " pod="ai-tenant-e2e-shared-a-544d30/auth-test-model-544d30-kserve-84f7bfc47f-2xgt6"
Jul 06 20:59:55.298635 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:59:55.298618    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb551609-543a-47a6-aad7-515f3001dd22-home\") pod \"auth-test-model-544d30-kserve-84f7bfc47f-2xgt6\" (UID: \"eb551609-543a-47a6-aad7-515f3001dd22\") " pod="ai-tenant-e2e-shared-a-544d30/auth-test-model-544d30-kserve-84f7bfc47f-2xgt6"
Jul 06 20:59:55.298672 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:59:55.298657    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb551609-543a-47a6-aad7-515f3001dd22-model-cache\") pod \"auth-test-model-544d30-kserve-84f7bfc47f-2xgt6\" (UID: \"eb551609-543a-47a6-aad7-515f3001dd22\") " pod="ai-tenant-e2e-shared-a-544d30/auth-test-model-544d30-kserve-84f7bfc47f-2xgt6"
Jul 06 20:59:55.300437 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:59:55.300413    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb551609-543a-47a6-aad7-515f3001dd22-dshm\") pod \"auth-test-model-544d30-kserve-84f7bfc47f-2xgt6\" (UID: \"eb551609-543a-47a6-aad7-515f3001dd22\") " pod="ai-tenant-e2e-shared-a-544d30/auth-test-model-544d30-kserve-84f7bfc47f-2xgt6"
Jul 06 20:59:55.300592 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:59:55.300564    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb551609-543a-47a6-aad7-515f3001dd22-tls-certs\") pod \"auth-test-model-544d30-kserve-84f7bfc47f-2xgt6\" (UID: \"eb551609-543a-47a6-aad7-515f3001dd22\") " pod="ai-tenant-e2e-shared-a-544d30/auth-test-model-544d30-kserve-84f7bfc47f-2xgt6"
Jul 06 20:59:55.307518 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:59:55.307495    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s25j5\" (UniqueName: \"kubernetes.io/projected/eb551609-543a-47a6-aad7-515f3001dd22-kube-api-access-s25j5\") pod \"auth-test-model-544d30-kserve-84f7bfc47f-2xgt6\" (UID: \"eb551609-543a-47a6-aad7-515f3001dd22\") " pod="ai-tenant-e2e-shared-a-544d30/auth-test-model-544d30-kserve-84f7bfc47f-2xgt6"
Jul 06 20:59:55.377334 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:59:55.377248    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-544d30/auth-test-model-544d30-kserve-84f7bfc47f-2xgt6"
Jul 06 20:59:55.552524 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:59:55.552498    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-544d30/auth-test-model-544d30-kserve-84f7bfc47f-2xgt6"]
Jul 06 20:59:55.554195 ip-10-0-141-123 kubenswrapper[2563]: W0706 20:59:55.554167    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeb551609_543a_47a6_aad7_515f3001dd22.slice/crio-3f41ab5018a5be49267fcc87266aad99b8e56b4ac0286983398c7fb4f5ae7aae WatchSource:0}: Error finding container 3f41ab5018a5be49267fcc87266aad99b8e56b4ac0286983398c7fb4f5ae7aae: Status 404 returned error can't find the container with id 3f41ab5018a5be49267fcc87266aad99b8e56b4ac0286983398c7fb4f5ae7aae
Jul 06 20:59:55.795156 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:59:55.795061    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-544d30/auth-test-model-544d30-kserve-84f7bfc47f-2xgt6" event={"ID":"eb551609-543a-47a6-aad7-515f3001dd22","Type":"ContainerStarted","Data":"88840562a7cdc07728b39cdf7ca132eda3abb87d62cdd39ffe20f5cc214c4d32"}
Jul 06 20:59:55.795156 ip-10-0-141-123 kubenswrapper[2563]: I0706 20:59:55.795110    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-544d30/auth-test-model-544d30-kserve-84f7bfc47f-2xgt6" event={"ID":"eb551609-543a-47a6-aad7-515f3001dd22","Type":"ContainerStarted","Data":"3f41ab5018a5be49267fcc87266aad99b8e56b4ac0286983398c7fb4f5ae7aae"}
Jul 06 21:00:01.190401 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:00:01.190369    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 21:00:01.313064 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:00:01.313026    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 21:00:05.300410 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:00:05.300330    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-544d30/auth-test-model-544d30-kserve-84f7bfc47f-2xgt6" podStartSLOduration=10.300307464 podStartE2EDuration="10.300307464s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:59:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:59:55.837128781 +0000 UTC m=+2159.553239443" watchObservedRunningTime="2026-07-06 21:00:05.300307464 +0000 UTC m=+2169.016418129"
Jul 06 21:00:05.302245 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:00:05.302225    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-clwd5"]
Jul 06 21:00:05.377992 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:00:05.377958    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-544d30/auth-test-model-544d30-kserve-84f7bfc47f-2xgt6"
Jul 06 21:00:05.377992 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:00:05.377999    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-544d30/auth-test-model-544d30-kserve-84f7bfc47f-2xgt6"
Jul 06 21:00:05.390964 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:00:05.390942    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-544d30/auth-test-model-544d30-kserve-84f7bfc47f-2xgt6"
Jul 06 21:00:05.855555 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:00:05.855528    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-544d30/auth-test-model-544d30-kserve-84f7bfc47f-2xgt6"
Jul 06 21:00:10.644214 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:00:10.644179    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 21:00:10.743635 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:00:10.743605    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 21:00:18.841828 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:00:18.841798    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-cd7c86/auth-test-model-cd7c86-kserve-5768594587-mjnnj"]
Jul 06 21:00:18.846248 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:00:18.846224    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-cd7c86/auth-test-model-cd7c86-kserve-5768594587-mjnnj"
Jul 06 21:00:18.850453 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:00:18.850431    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-cd7c86\"/\"default-dockercfg-wqfc5\""
Jul 06 21:00:18.851298 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:00:18.851275    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-cd7c86\"/\"openshift-service-ca.crt\""
Jul 06 21:00:18.851298 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:00:18.851279    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-cd7c86\"/\"auth-test-model-cd7c86-kserve-self-signed-certs\""
Jul 06 21:00:18.851484 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:00:18.851320    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-cd7c86\"/\"kube-root-ca.crt\""
Jul 06 21:00:18.858036 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:00:18.858015    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-cd7c86/auth-test-model-cd7c86-kserve-5768594587-mjnnj"]
Jul 06 21:00:18.879400 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:00:18.879377    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 21:00:18.904861 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:00:18.904841    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0464cc88-2557-4439-99ea-c685b35ed7a5-tmp-dir\") pod \"auth-test-model-cd7c86-kserve-5768594587-mjnnj\" (UID: \"0464cc88-2557-4439-99ea-c685b35ed7a5\") " pod="ai-tenant-e2e-shared-b-cd7c86/auth-test-model-cd7c86-kserve-5768594587-mjnnj"
Jul 06 21:00:18.904950 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:00:18.904869    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0464cc88-2557-4439-99ea-c685b35ed7a5-home\") pod \"auth-test-model-cd7c86-kserve-5768594587-mjnnj\" (UID: \"0464cc88-2557-4439-99ea-c685b35ed7a5\") " pod="ai-tenant-e2e-shared-b-cd7c86/auth-test-model-cd7c86-kserve-5768594587-mjnnj"
Jul 06 21:00:18.904950 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:00:18.904893    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0464cc88-2557-4439-99ea-c685b35ed7a5-model-cache\") pod \"auth-test-model-cd7c86-kserve-5768594587-mjnnj\" (UID: \"0464cc88-2557-4439-99ea-c685b35ed7a5\") " pod="ai-tenant-e2e-shared-b-cd7c86/auth-test-model-cd7c86-kserve-5768594587-mjnnj"
Jul 06 21:00:18.905022 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:00:18.904982    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0464cc88-2557-4439-99ea-c685b35ed7a5-tls-certs\") pod \"auth-test-model-cd7c86-kserve-5768594587-mjnnj\" (UID: \"0464cc88-2557-4439-99ea-c685b35ed7a5\") " pod="ai-tenant-e2e-shared-b-cd7c86/auth-test-model-cd7c86-kserve-5768594587-mjnnj"
Jul 06 21:00:18.905022 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:00:18.905011    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0464cc88-2557-4439-99ea-c685b35ed7a5-dshm\") pod \"auth-test-model-cd7c86-kserve-5768594587-mjnnj\" (UID: \"0464cc88-2557-4439-99ea-c685b35ed7a5\") " pod="ai-tenant-e2e-shared-b-cd7c86/auth-test-model-cd7c86-kserve-5768594587-mjnnj"
Jul 06 21:00:18.905100 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:00:18.905091    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5d5gs\" (UniqueName: \"kubernetes.io/projected/0464cc88-2557-4439-99ea-c685b35ed7a5-kube-api-access-5d5gs\") pod \"auth-test-model-cd7c86-kserve-5768594587-mjnnj\" (UID: \"0464cc88-2557-4439-99ea-c685b35ed7a5\") " pod="ai-tenant-e2e-shared-b-cd7c86/auth-test-model-cd7c86-kserve-5768594587-mjnnj"
Jul 06 21:00:19.005928 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:00:19.005900    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0464cc88-2557-4439-99ea-c685b35ed7a5-tmp-dir\") pod \"auth-test-model-cd7c86-kserve-5768594587-mjnnj\" (UID: \"0464cc88-2557-4439-99ea-c685b35ed7a5\") " pod="ai-tenant-e2e-shared-b-cd7c86/auth-test-model-cd7c86-kserve-5768594587-mjnnj"
Jul 06 21:00:19.006056 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:00:19.005935    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0464cc88-2557-4439-99ea-c685b35ed7a5-home\") pod \"auth-test-model-cd7c86-kserve-5768594587-mjnnj\" (UID: \"0464cc88-2557-4439-99ea-c685b35ed7a5\") " pod="ai-tenant-e2e-shared-b-cd7c86/auth-test-model-cd7c86-kserve-5768594587-mjnnj"
Jul 06 21:00:19.006056 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:00:19.005967    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0464cc88-2557-4439-99ea-c685b35ed7a5-model-cache\") pod \"auth-test-model-cd7c86-kserve-5768594587-mjnnj\" (UID: \"0464cc88-2557-4439-99ea-c685b35ed7a5\") " pod="ai-tenant-e2e-shared-b-cd7c86/auth-test-model-cd7c86-kserve-5768594587-mjnnj"
Jul 06 21:00:19.006056 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:00:19.006022    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0464cc88-2557-4439-99ea-c685b35ed7a5-tls-certs\") pod \"auth-test-model-cd7c86-kserve-5768594587-mjnnj\" (UID: \"0464cc88-2557-4439-99ea-c685b35ed7a5\") " pod="ai-tenant-e2e-shared-b-cd7c86/auth-test-model-cd7c86-kserve-5768594587-mjnnj"
Jul 06 21:00:19.006056 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:00:19.006053    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0464cc88-2557-4439-99ea-c685b35ed7a5-dshm\") pod \"auth-test-model-cd7c86-kserve-5768594587-mjnnj\" (UID: \"0464cc88-2557-4439-99ea-c685b35ed7a5\") " pod="ai-tenant-e2e-shared-b-cd7c86/auth-test-model-cd7c86-kserve-5768594587-mjnnj"
Jul 06 21:00:19.006277 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:00:19.006128    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5d5gs\" (UniqueName: \"kubernetes.io/projected/0464cc88-2557-4439-99ea-c685b35ed7a5-kube-api-access-5d5gs\") pod \"auth-test-model-cd7c86-kserve-5768594587-mjnnj\" (UID: \"0464cc88-2557-4439-99ea-c685b35ed7a5\") " pod="ai-tenant-e2e-shared-b-cd7c86/auth-test-model-cd7c86-kserve-5768594587-mjnnj"
Jul 06 21:00:19.006384 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:00:19.006360    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0464cc88-2557-4439-99ea-c685b35ed7a5-home\") pod \"auth-test-model-cd7c86-kserve-5768594587-mjnnj\" (UID: \"0464cc88-2557-4439-99ea-c685b35ed7a5\") " pod="ai-tenant-e2e-shared-b-cd7c86/auth-test-model-cd7c86-kserve-5768594587-mjnnj"
Jul 06 21:00:19.006449 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:00:19.006386    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0464cc88-2557-4439-99ea-c685b35ed7a5-model-cache\") pod \"auth-test-model-cd7c86-kserve-5768594587-mjnnj\" (UID: \"0464cc88-2557-4439-99ea-c685b35ed7a5\") " pod="ai-tenant-e2e-shared-b-cd7c86/auth-test-model-cd7c86-kserve-5768594587-mjnnj"
Jul 06 21:00:19.006449 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:00:19.006429    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0464cc88-2557-4439-99ea-c685b35ed7a5-tmp-dir\") pod \"auth-test-model-cd7c86-kserve-5768594587-mjnnj\" (UID: \"0464cc88-2557-4439-99ea-c685b35ed7a5\") " pod="ai-tenant-e2e-shared-b-cd7c86/auth-test-model-cd7c86-kserve-5768594587-mjnnj"
Jul 06 21:00:19.008334 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:00:19.008317    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0464cc88-2557-4439-99ea-c685b35ed7a5-dshm\") pod \"auth-test-model-cd7c86-kserve-5768594587-mjnnj\" (UID: \"0464cc88-2557-4439-99ea-c685b35ed7a5\") " pod="ai-tenant-e2e-shared-b-cd7c86/auth-test-model-cd7c86-kserve-5768594587-mjnnj"
Jul 06 21:00:19.008502 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:00:19.008485    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0464cc88-2557-4439-99ea-c685b35ed7a5-tls-certs\") pod \"auth-test-model-cd7c86-kserve-5768594587-mjnnj\" (UID: \"0464cc88-2557-4439-99ea-c685b35ed7a5\") " pod="ai-tenant-e2e-shared-b-cd7c86/auth-test-model-cd7c86-kserve-5768594587-mjnnj"
Jul 06 21:00:19.014936 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:00:19.014913    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5d5gs\" (UniqueName: \"kubernetes.io/projected/0464cc88-2557-4439-99ea-c685b35ed7a5-kube-api-access-5d5gs\") pod \"auth-test-model-cd7c86-kserve-5768594587-mjnnj\" (UID: \"0464cc88-2557-4439-99ea-c685b35ed7a5\") " pod="ai-tenant-e2e-shared-b-cd7c86/auth-test-model-cd7c86-kserve-5768594587-mjnnj"
Jul 06 21:00:19.157791 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:00:19.157688    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-cd7c86/auth-test-model-cd7c86-kserve-5768594587-mjnnj"
Jul 06 21:00:19.295684 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:00:19.295658    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-cd7c86/auth-test-model-cd7c86-kserve-5768594587-mjnnj"]
Jul 06 21:00:19.296053 ip-10-0-141-123 kubenswrapper[2563]: W0706 21:00:19.296029    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0464cc88_2557_4439_99ea_c685b35ed7a5.slice/crio-0fdd831147ab089c1ea83aaee1d004ca354372c0b4802fce0d90590276899200 WatchSource:0}: Error finding container 0fdd831147ab089c1ea83aaee1d004ca354372c0b4802fce0d90590276899200: Status 404 returned error can't find the container with id 0fdd831147ab089c1ea83aaee1d004ca354372c0b4802fce0d90590276899200
Jul 06 21:00:19.904965 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:00:19.904937    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-cd7c86/auth-test-model-cd7c86-kserve-5768594587-mjnnj" event={"ID":"0464cc88-2557-4439-99ea-c685b35ed7a5","Type":"ContainerStarted","Data":"733e77d35d9cd4ad038e5a2eed76cbaa9e577e3cc98fe6a7e2a368faf7a16512"}
Jul 06 21:00:19.905363 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:00:19.904976    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-cd7c86/auth-test-model-cd7c86-kserve-5768594587-mjnnj" event={"ID":"0464cc88-2557-4439-99ea-c685b35ed7a5","Type":"ContainerStarted","Data":"0fdd831147ab089c1ea83aaee1d004ca354372c0b4802fce0d90590276899200"}
Jul 06 21:00:19.925965 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:00:19.925911    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-cd7c86/auth-test-model-cd7c86-kserve-5768594587-mjnnj" podStartSLOduration=1.9258930570000001 podStartE2EDuration="1.925893057s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:00:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:00:19.925223634 +0000 UTC m=+2183.641334295" watchObservedRunningTime="2026-07-06 21:00:19.925893057 +0000 UTC m=+2183.642003718"
Jul 06 21:00:22.797627 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:00:22.797593    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-clwd5"]
Jul 06 21:00:29.158069 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:00:29.158032    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-cd7c86/auth-test-model-cd7c86-kserve-5768594587-mjnnj"
Jul 06 21:00:29.158560 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:00:29.158323    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-cd7c86/auth-test-model-cd7c86-kserve-5768594587-mjnnj"
Jul 06 21:00:29.171060 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:00:29.171035    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-cd7c86/auth-test-model-cd7c86-kserve-5768594587-mjnnj"
Jul 06 21:00:29.962012 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:00:29.961985    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-cd7c86/auth-test-model-cd7c86-kserve-5768594587-mjnnj"
Jul 06 21:00:37.842572 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:00:37.842542    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-clwd5"]
Jul 06 21:00:44.613437 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:00:44.613402    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 21:00:44.714533 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:00:44.714501    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 21:00:54.788384 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:00:54.788291    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-clwd5"]
Jul 06 21:00:55.313632 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:00:55.313592    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 21:01:09.001809 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:01:09.001773    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-clwd5"]
Jul 06 21:01:20.757927 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:01:20.757819    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 21:01:20.853429 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:01:20.853399    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 21:01:26.618746 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:01:26.618703    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-clwd5"]
Jul 06 21:01:43.501919 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:01:43.501884    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-clwd5"]
Jul 06 21:01:59.699365 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:01:59.699327    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-clwd5"]
Jul 06 21:02:15.193866 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:02:15.193831    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-clwd5"]
Jul 06 21:02:25.157990 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:02:25.157954    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-544d30/auth-test-model-544d30-kserve-84f7bfc47f-2xgt6"]
Jul 06 21:02:25.181715 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:02:25.181677    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 21:02:26.072701 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:02:26.072659    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-cd7c86/auth-test-model-cd7c86-kserve-5768594587-mjnnj"]
Jul 06 21:02:26.083490 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:02:26.083461    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 21:02:28.210282 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:02:28.210247    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-clwd5"]
Jul 06 21:02:33.153339 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:02:33.153287    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 21:02:40.178443 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:02:40.178403    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-544d30/auth-test-model-544d30-kserve-84f7bfc47f-2xgt6" podUID="eb551609-543a-47a6-aad7-515f3001dd22" containerName="main" containerID="cri-o://88840562a7cdc07728b39cdf7ca132eda3abb87d62cdd39ffe20f5cc214c4d32" gracePeriod=45
Jul 06 21:02:40.424618 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:02:40.424593    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-544d30/auth-test-model-544d30-kserve-84f7bfc47f-2xgt6"
Jul 06 21:02:40.506693 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:02:40.506610    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb551609-543a-47a6-aad7-515f3001dd22-dshm\") pod \"eb551609-543a-47a6-aad7-515f3001dd22\" (UID: \"eb551609-543a-47a6-aad7-515f3001dd22\") "
Jul 06 21:02:40.506879 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:02:40.506753    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb551609-543a-47a6-aad7-515f3001dd22-tmp-dir\") pod \"eb551609-543a-47a6-aad7-515f3001dd22\" (UID: \"eb551609-543a-47a6-aad7-515f3001dd22\") "
Jul 06 21:02:40.506879 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:02:40.506790    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb551609-543a-47a6-aad7-515f3001dd22-model-cache\") pod \"eb551609-543a-47a6-aad7-515f3001dd22\" (UID: \"eb551609-543a-47a6-aad7-515f3001dd22\") "
Jul 06 21:02:40.506879 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:02:40.506843    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s25j5\" (UniqueName: \"kubernetes.io/projected/eb551609-543a-47a6-aad7-515f3001dd22-kube-api-access-s25j5\") pod \"eb551609-543a-47a6-aad7-515f3001dd22\" (UID: \"eb551609-543a-47a6-aad7-515f3001dd22\") "
Jul 06 21:02:40.507048 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:02:40.506881    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb551609-543a-47a6-aad7-515f3001dd22-home\") pod \"eb551609-543a-47a6-aad7-515f3001dd22\" (UID: \"eb551609-543a-47a6-aad7-515f3001dd22\") "
Jul 06 21:02:40.507048 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:02:40.506906    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb551609-543a-47a6-aad7-515f3001dd22-tls-certs\") pod \"eb551609-543a-47a6-aad7-515f3001dd22\" (UID: \"eb551609-543a-47a6-aad7-515f3001dd22\") "
Jul 06 21:02:40.507048 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:02:40.507016    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb551609-543a-47a6-aad7-515f3001dd22-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "eb551609-543a-47a6-aad7-515f3001dd22" (UID: "eb551609-543a-47a6-aad7-515f3001dd22"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:02:40.507210 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:02:40.507088    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb551609-543a-47a6-aad7-515f3001dd22-model-cache" (OuterVolumeSpecName: "model-cache") pod "eb551609-543a-47a6-aad7-515f3001dd22" (UID: "eb551609-543a-47a6-aad7-515f3001dd22"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:02:40.507210 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:02:40.507102    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb551609-543a-47a6-aad7-515f3001dd22-home" (OuterVolumeSpecName: "home") pod "eb551609-543a-47a6-aad7-515f3001dd22" (UID: "eb551609-543a-47a6-aad7-515f3001dd22"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:02:40.507294 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:02:40.507234    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb551609-543a-47a6-aad7-515f3001dd22-home\") on node \"ip-10-0-141-123.ec2.internal\" DevicePath \"\""
Jul 06 21:02:40.507294 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:02:40.507254    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb551609-543a-47a6-aad7-515f3001dd22-tmp-dir\") on node \"ip-10-0-141-123.ec2.internal\" DevicePath \"\""
Jul 06 21:02:40.507294 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:02:40.507269    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb551609-543a-47a6-aad7-515f3001dd22-model-cache\") on node \"ip-10-0-141-123.ec2.internal\" DevicePath \"\""
Jul 06 21:02:40.508994 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:02:40.508962    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb551609-543a-47a6-aad7-515f3001dd22-dshm" (OuterVolumeSpecName: "dshm") pod "eb551609-543a-47a6-aad7-515f3001dd22" (UID: "eb551609-543a-47a6-aad7-515f3001dd22"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:02:40.508994 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:02:40.508969    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb551609-543a-47a6-aad7-515f3001dd22-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "eb551609-543a-47a6-aad7-515f3001dd22" (UID: "eb551609-543a-47a6-aad7-515f3001dd22"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 21:02:40.509170 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:02:40.509023    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb551609-543a-47a6-aad7-515f3001dd22-kube-api-access-s25j5" (OuterVolumeSpecName: "kube-api-access-s25j5") pod "eb551609-543a-47a6-aad7-515f3001dd22" (UID: "eb551609-543a-47a6-aad7-515f3001dd22"). InnerVolumeSpecName "kube-api-access-s25j5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 21:02:40.579388 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:02:40.579350    2563 generic.go:356] "Generic (PLEG): container finished" podID="eb551609-543a-47a6-aad7-515f3001dd22" containerID="88840562a7cdc07728b39cdf7ca132eda3abb87d62cdd39ffe20f5cc214c4d32" exitCode=0
Jul 06 21:02:40.579533 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:02:40.579412    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-544d30/auth-test-model-544d30-kserve-84f7bfc47f-2xgt6"
Jul 06 21:02:40.579533 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:02:40.579435    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-544d30/auth-test-model-544d30-kserve-84f7bfc47f-2xgt6" event={"ID":"eb551609-543a-47a6-aad7-515f3001dd22","Type":"ContainerDied","Data":"88840562a7cdc07728b39cdf7ca132eda3abb87d62cdd39ffe20f5cc214c4d32"}
Jul 06 21:02:40.579533 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:02:40.579482    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-544d30/auth-test-model-544d30-kserve-84f7bfc47f-2xgt6" event={"ID":"eb551609-543a-47a6-aad7-515f3001dd22","Type":"ContainerDied","Data":"3f41ab5018a5be49267fcc87266aad99b8e56b4ac0286983398c7fb4f5ae7aae"}
Jul 06 21:02:40.579533 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:02:40.579500    2563 scope.go:117] "RemoveContainer" containerID="88840562a7cdc07728b39cdf7ca132eda3abb87d62cdd39ffe20f5cc214c4d32"
Jul 06 21:02:40.589369 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:02:40.589349    2563 scope.go:117] "RemoveContainer" containerID="88840562a7cdc07728b39cdf7ca132eda3abb87d62cdd39ffe20f5cc214c4d32"
Jul 06 21:02:40.589676 ip-10-0-141-123 kubenswrapper[2563]: E0706 21:02:40.589655    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88840562a7cdc07728b39cdf7ca132eda3abb87d62cdd39ffe20f5cc214c4d32\": container with ID starting with 88840562a7cdc07728b39cdf7ca132eda3abb87d62cdd39ffe20f5cc214c4d32 not found: ID does not exist" containerID="88840562a7cdc07728b39cdf7ca132eda3abb87d62cdd39ffe20f5cc214c4d32"
Jul 06 21:02:40.589773 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:02:40.589685    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88840562a7cdc07728b39cdf7ca132eda3abb87d62cdd39ffe20f5cc214c4d32"} err="failed to get container status \"88840562a7cdc07728b39cdf7ca132eda3abb87d62cdd39ffe20f5cc214c4d32\": rpc error: code = NotFound desc = could not find container \"88840562a7cdc07728b39cdf7ca132eda3abb87d62cdd39ffe20f5cc214c4d32\": container with ID starting with 88840562a7cdc07728b39cdf7ca132eda3abb87d62cdd39ffe20f5cc214c4d32 not found: ID does not exist"
Jul 06 21:02:40.607877 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:02:40.607844    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s25j5\" (UniqueName: \"kubernetes.io/projected/eb551609-543a-47a6-aad7-515f3001dd22-kube-api-access-s25j5\") on node \"ip-10-0-141-123.ec2.internal\" DevicePath \"\""
Jul 06 21:02:40.607877 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:02:40.607873    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb551609-543a-47a6-aad7-515f3001dd22-tls-certs\") on node \"ip-10-0-141-123.ec2.internal\" DevicePath \"\""
Jul 06 21:02:40.607877 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:02:40.607882    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb551609-543a-47a6-aad7-515f3001dd22-dshm\") on node \"ip-10-0-141-123.ec2.internal\" DevicePath \"\""
Jul 06 21:02:40.609199 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:02:40.609178    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-544d30/auth-test-model-544d30-kserve-84f7bfc47f-2xgt6"]
Jul 06 21:02:40.615974 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:02:40.615948    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-544d30/auth-test-model-544d30-kserve-84f7bfc47f-2xgt6"]
Jul 06 21:02:40.640087 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:02:40.640053    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 21:02:40.834570 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:02:40.834538    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb551609-543a-47a6-aad7-515f3001dd22" path="/var/lib/kubelet/pods/eb551609-543a-47a6-aad7-515f3001dd22/volumes"
Jul 06 21:02:41.093181 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:02:41.093095    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-cd7c86/auth-test-model-cd7c86-kserve-5768594587-mjnnj" podUID="0464cc88-2557-4439-99ea-c685b35ed7a5" containerName="main" containerID="cri-o://733e77d35d9cd4ad038e5a2eed76cbaa9e577e3cc98fe6a7e2a368faf7a16512" gracePeriod=45
Jul 06 21:02:41.344894 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:02:41.344838    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-cd7c86/auth-test-model-cd7c86-kserve-5768594587-mjnnj"
Jul 06 21:02:41.414849 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:02:41.414815    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0464cc88-2557-4439-99ea-c685b35ed7a5-tls-certs\") pod \"0464cc88-2557-4439-99ea-c685b35ed7a5\" (UID: \"0464cc88-2557-4439-99ea-c685b35ed7a5\") "
Jul 06 21:02:41.415029 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:02:41.414857    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0464cc88-2557-4439-99ea-c685b35ed7a5-tmp-dir\") pod \"0464cc88-2557-4439-99ea-c685b35ed7a5\" (UID: \"0464cc88-2557-4439-99ea-c685b35ed7a5\") "
Jul 06 21:02:41.415029 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:02:41.414901    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5d5gs\" (UniqueName: \"kubernetes.io/projected/0464cc88-2557-4439-99ea-c685b35ed7a5-kube-api-access-5d5gs\") pod \"0464cc88-2557-4439-99ea-c685b35ed7a5\" (UID: \"0464cc88-2557-4439-99ea-c685b35ed7a5\") "
Jul 06 21:02:41.415029 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:02:41.414930    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0464cc88-2557-4439-99ea-c685b35ed7a5-dshm\") pod \"0464cc88-2557-4439-99ea-c685b35ed7a5\" (UID: \"0464cc88-2557-4439-99ea-c685b35ed7a5\") "
Jul 06 21:02:41.415029 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:02:41.414969    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0464cc88-2557-4439-99ea-c685b35ed7a5-home\") pod \"0464cc88-2557-4439-99ea-c685b35ed7a5\" (UID: \"0464cc88-2557-4439-99ea-c685b35ed7a5\") "
Jul 06 21:02:41.415029 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:02:41.414993    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0464cc88-2557-4439-99ea-c685b35ed7a5-model-cache\") pod \"0464cc88-2557-4439-99ea-c685b35ed7a5\" (UID: \"0464cc88-2557-4439-99ea-c685b35ed7a5\") "
Jul 06 21:02:41.415306 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:02:41.415236    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0464cc88-2557-4439-99ea-c685b35ed7a5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0464cc88-2557-4439-99ea-c685b35ed7a5" (UID: "0464cc88-2557-4439-99ea-c685b35ed7a5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:02:41.415306 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:02:41.415251    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0464cc88-2557-4439-99ea-c685b35ed7a5-home" (OuterVolumeSpecName: "home") pod "0464cc88-2557-4439-99ea-c685b35ed7a5" (UID: "0464cc88-2557-4439-99ea-c685b35ed7a5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:02:41.415390 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:02:41.415320    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0464cc88-2557-4439-99ea-c685b35ed7a5-model-cache" (OuterVolumeSpecName: "model-cache") pod "0464cc88-2557-4439-99ea-c685b35ed7a5" (UID: "0464cc88-2557-4439-99ea-c685b35ed7a5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:02:41.417154 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:02:41.417133    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0464cc88-2557-4439-99ea-c685b35ed7a5-dshm" (OuterVolumeSpecName: "dshm") pod "0464cc88-2557-4439-99ea-c685b35ed7a5" (UID: "0464cc88-2557-4439-99ea-c685b35ed7a5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:02:41.417230 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:02:41.417205    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0464cc88-2557-4439-99ea-c685b35ed7a5-kube-api-access-5d5gs" (OuterVolumeSpecName: "kube-api-access-5d5gs") pod "0464cc88-2557-4439-99ea-c685b35ed7a5" (UID: "0464cc88-2557-4439-99ea-c685b35ed7a5"). InnerVolumeSpecName "kube-api-access-5d5gs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 21:02:41.417280 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:02:41.417264    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0464cc88-2557-4439-99ea-c685b35ed7a5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0464cc88-2557-4439-99ea-c685b35ed7a5" (UID: "0464cc88-2557-4439-99ea-c685b35ed7a5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 21:02:41.515822 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:02:41.515783    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0464cc88-2557-4439-99ea-c685b35ed7a5-tls-certs\") on node \"ip-10-0-141-123.ec2.internal\" DevicePath \"\""
Jul 06 21:02:41.515994 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:02:41.515836    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0464cc88-2557-4439-99ea-c685b35ed7a5-tmp-dir\") on node \"ip-10-0-141-123.ec2.internal\" DevicePath \"\""
Jul 06 21:02:41.515994 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:02:41.515850    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5d5gs\" (UniqueName: \"kubernetes.io/projected/0464cc88-2557-4439-99ea-c685b35ed7a5-kube-api-access-5d5gs\") on node \"ip-10-0-141-123.ec2.internal\" DevicePath \"\""
Jul 06 21:02:41.515994 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:02:41.515863    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0464cc88-2557-4439-99ea-c685b35ed7a5-dshm\") on node \"ip-10-0-141-123.ec2.internal\" DevicePath \"\""
Jul 06 21:02:41.515994 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:02:41.515875    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0464cc88-2557-4439-99ea-c685b35ed7a5-home\") on node \"ip-10-0-141-123.ec2.internal\" DevicePath \"\""
Jul 06 21:02:41.515994 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:02:41.515885    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0464cc88-2557-4439-99ea-c685b35ed7a5-model-cache\") on node \"ip-10-0-141-123.ec2.internal\" DevicePath \"\""
Jul 06 21:02:41.586459 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:02:41.586425    2563 generic.go:356] "Generic (PLEG): container finished" podID="0464cc88-2557-4439-99ea-c685b35ed7a5" containerID="733e77d35d9cd4ad038e5a2eed76cbaa9e577e3cc98fe6a7e2a368faf7a16512" exitCode=0
Jul 06 21:02:41.586459 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:02:41.586461    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-cd7c86/auth-test-model-cd7c86-kserve-5768594587-mjnnj" event={"ID":"0464cc88-2557-4439-99ea-c685b35ed7a5","Type":"ContainerDied","Data":"733e77d35d9cd4ad038e5a2eed76cbaa9e577e3cc98fe6a7e2a368faf7a16512"}
Jul 06 21:02:41.586698 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:02:41.586470    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-cd7c86/auth-test-model-cd7c86-kserve-5768594587-mjnnj"
Jul 06 21:02:41.586698 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:02:41.586487    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-cd7c86/auth-test-model-cd7c86-kserve-5768594587-mjnnj" event={"ID":"0464cc88-2557-4439-99ea-c685b35ed7a5","Type":"ContainerDied","Data":"0fdd831147ab089c1ea83aaee1d004ca354372c0b4802fce0d90590276899200"}
Jul 06 21:02:41.586698 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:02:41.586503    2563 scope.go:117] "RemoveContainer" containerID="733e77d35d9cd4ad038e5a2eed76cbaa9e577e3cc98fe6a7e2a368faf7a16512"
Jul 06 21:02:41.595652 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:02:41.595634    2563 scope.go:117] "RemoveContainer" containerID="733e77d35d9cd4ad038e5a2eed76cbaa9e577e3cc98fe6a7e2a368faf7a16512"
Jul 06 21:02:41.595917 ip-10-0-141-123 kubenswrapper[2563]: E0706 21:02:41.595898    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"733e77d35d9cd4ad038e5a2eed76cbaa9e577e3cc98fe6a7e2a368faf7a16512\": container with ID starting with 733e77d35d9cd4ad038e5a2eed76cbaa9e577e3cc98fe6a7e2a368faf7a16512 not found: ID does not exist" containerID="733e77d35d9cd4ad038e5a2eed76cbaa9e577e3cc98fe6a7e2a368faf7a16512"
Jul 06 21:02:41.595991 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:02:41.595924    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"733e77d35d9cd4ad038e5a2eed76cbaa9e577e3cc98fe6a7e2a368faf7a16512"} err="failed to get container status \"733e77d35d9cd4ad038e5a2eed76cbaa9e577e3cc98fe6a7e2a368faf7a16512\": rpc error: code = NotFound desc = could not find container \"733e77d35d9cd4ad038e5a2eed76cbaa9e577e3cc98fe6a7e2a368faf7a16512\": container with ID starting with 733e77d35d9cd4ad038e5a2eed76cbaa9e577e3cc98fe6a7e2a368faf7a16512 not found: ID does not exist"
Jul 06 21:02:41.608649 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:02:41.608613    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-cd7c86/auth-test-model-cd7c86-kserve-5768594587-mjnnj"]
Jul 06 21:02:41.614515 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:02:41.614490    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-cd7c86/auth-test-model-cd7c86-kserve-5768594587-mjnnj"]
Jul 06 21:02:42.492061 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:02:42.492030    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-clwd5"]
Jul 06 21:02:42.834393 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:02:42.834353    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0464cc88-2557-4439-99ea-c685b35ed7a5" path="/var/lib/kubelet/pods/0464cc88-2557-4439-99ea-c685b35ed7a5/volumes"
Jul 06 21:02:58.952540 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:02:58.952505    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-cd7c86/sub-test-model-cd7c86-kserve-84bf47c64f-s52pp"]
Jul 06 21:02:58.952993 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:02:58.952916    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0464cc88-2557-4439-99ea-c685b35ed7a5" containerName="main"
Jul 06 21:02:58.952993 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:02:58.952928    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0464cc88-2557-4439-99ea-c685b35ed7a5" containerName="main"
Jul 06 21:02:58.952993 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:02:58.952941    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb551609-543a-47a6-aad7-515f3001dd22" containerName="main"
Jul 06 21:02:58.952993 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:02:58.952947    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb551609-543a-47a6-aad7-515f3001dd22" containerName="main"
Jul 06 21:02:58.953129 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:02:58.953017    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb551609-543a-47a6-aad7-515f3001dd22" containerName="main"
Jul 06 21:02:58.953129 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:02:58.953027    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="0464cc88-2557-4439-99ea-c685b35ed7a5" containerName="main"
Jul 06 21:02:58.957704 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:02:58.957684    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-cd7c86/sub-test-model-cd7c86-kserve-84bf47c64f-s52pp"
Jul 06 21:02:58.960481 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:02:58.960457    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-cd7c86\"/\"kube-root-ca.crt\""
Jul 06 21:02:58.961090 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:02:58.961071    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-cd7c86\"/\"default-dockercfg-wqfc5\""
Jul 06 21:02:58.961213 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:02:58.961158    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-cd7c86\"/\"sub-test-model-cd7c86-kserve-self-signed-certs\""
Jul 06 21:02:58.961213 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:02:58.961183    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-cd7c86\"/\"openshift-service-ca.crt\""
Jul 06 21:02:58.975668 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:02:58.975637    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-cd7c86/sub-test-model-cd7c86-kserve-84bf47c64f-s52pp"]
Jul 06 21:02:58.989306 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:02:58.989274    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 21:02:59.083904 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:02:59.083872    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/241c07fa-93e7-4167-ac56-60eb6807a16b-tmp-dir\") pod \"sub-test-model-cd7c86-kserve-84bf47c64f-s52pp\" (UID: \"241c07fa-93e7-4167-ac56-60eb6807a16b\") " pod="ai-tenant-e2e-shared-b-cd7c86/sub-test-model-cd7c86-kserve-84bf47c64f-s52pp"
Jul 06 21:02:59.084077 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:02:59.083936    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/241c07fa-93e7-4167-ac56-60eb6807a16b-home\") pod \"sub-test-model-cd7c86-kserve-84bf47c64f-s52pp\" (UID: \"241c07fa-93e7-4167-ac56-60eb6807a16b\") " pod="ai-tenant-e2e-shared-b-cd7c86/sub-test-model-cd7c86-kserve-84bf47c64f-s52pp"
Jul 06 21:02:59.084077 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:02:59.083963    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/241c07fa-93e7-4167-ac56-60eb6807a16b-tls-certs\") pod \"sub-test-model-cd7c86-kserve-84bf47c64f-s52pp\" (UID: \"241c07fa-93e7-4167-ac56-60eb6807a16b\") " pod="ai-tenant-e2e-shared-b-cd7c86/sub-test-model-cd7c86-kserve-84bf47c64f-s52pp"
Jul 06 21:02:59.084077 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:02:59.083981    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/241c07fa-93e7-4167-ac56-60eb6807a16b-dshm\") pod \"sub-test-model-cd7c86-kserve-84bf47c64f-s52pp\" (UID: \"241c07fa-93e7-4167-ac56-60eb6807a16b\") " pod="ai-tenant-e2e-shared-b-cd7c86/sub-test-model-cd7c86-kserve-84bf47c64f-s52pp"
Jul 06 21:02:59.084077 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:02:59.083995    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/241c07fa-93e7-4167-ac56-60eb6807a16b-model-cache\") pod \"sub-test-model-cd7c86-kserve-84bf47c64f-s52pp\" (UID: \"241c07fa-93e7-4167-ac56-60eb6807a16b\") " pod="ai-tenant-e2e-shared-b-cd7c86/sub-test-model-cd7c86-kserve-84bf47c64f-s52pp"
Jul 06 21:02:59.084077 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:02:59.084011    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fdjv\" (UniqueName: \"kubernetes.io/projected/241c07fa-93e7-4167-ac56-60eb6807a16b-kube-api-access-9fdjv\") pod \"sub-test-model-cd7c86-kserve-84bf47c64f-s52pp\" (UID: \"241c07fa-93e7-4167-ac56-60eb6807a16b\") " pod="ai-tenant-e2e-shared-b-cd7c86/sub-test-model-cd7c86-kserve-84bf47c64f-s52pp"
Jul 06 21:02:59.184820 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:02:59.184785    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/241c07fa-93e7-4167-ac56-60eb6807a16b-tmp-dir\") pod \"sub-test-model-cd7c86-kserve-84bf47c64f-s52pp\" (UID: \"241c07fa-93e7-4167-ac56-60eb6807a16b\") " pod="ai-tenant-e2e-shared-b-cd7c86/sub-test-model-cd7c86-kserve-84bf47c64f-s52pp"
Jul 06 21:02:59.185015 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:02:59.184853    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/241c07fa-93e7-4167-ac56-60eb6807a16b-home\") pod \"sub-test-model-cd7c86-kserve-84bf47c64f-s52pp\" (UID: \"241c07fa-93e7-4167-ac56-60eb6807a16b\") " pod="ai-tenant-e2e-shared-b-cd7c86/sub-test-model-cd7c86-kserve-84bf47c64f-s52pp"
Jul 06 21:02:59.185015 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:02:59.184876    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/241c07fa-93e7-4167-ac56-60eb6807a16b-tls-certs\") pod \"sub-test-model-cd7c86-kserve-84bf47c64f-s52pp\" (UID: \"241c07fa-93e7-4167-ac56-60eb6807a16b\") " pod="ai-tenant-e2e-shared-b-cd7c86/sub-test-model-cd7c86-kserve-84bf47c64f-s52pp"
Jul 06 21:02:59.185015 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:02:59.184895    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/241c07fa-93e7-4167-ac56-60eb6807a16b-dshm\") pod \"sub-test-model-cd7c86-kserve-84bf47c64f-s52pp\" (UID: \"241c07fa-93e7-4167-ac56-60eb6807a16b\") " pod="ai-tenant-e2e-shared-b-cd7c86/sub-test-model-cd7c86-kserve-84bf47c64f-s52pp"
Jul 06 21:02:59.185015 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:02:59.184917    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/241c07fa-93e7-4167-ac56-60eb6807a16b-model-cache\") pod \"sub-test-model-cd7c86-kserve-84bf47c64f-s52pp\" (UID: \"241c07fa-93e7-4167-ac56-60eb6807a16b\") " pod="ai-tenant-e2e-shared-b-cd7c86/sub-test-model-cd7c86-kserve-84bf47c64f-s52pp"
Jul 06 21:02:59.185015 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:02:59.184937    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fdjv\" (UniqueName: \"kubernetes.io/projected/241c07fa-93e7-4167-ac56-60eb6807a16b-kube-api-access-9fdjv\") pod \"sub-test-model-cd7c86-kserve-84bf47c64f-s52pp\" (UID: \"241c07fa-93e7-4167-ac56-60eb6807a16b\") " pod="ai-tenant-e2e-shared-b-cd7c86/sub-test-model-cd7c86-kserve-84bf47c64f-s52pp"
Jul 06 21:02:59.185309 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:02:59.185217    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/241c07fa-93e7-4167-ac56-60eb6807a16b-tmp-dir\") pod \"sub-test-model-cd7c86-kserve-84bf47c64f-s52pp\" (UID: \"241c07fa-93e7-4167-ac56-60eb6807a16b\") " pod="ai-tenant-e2e-shared-b-cd7c86/sub-test-model-cd7c86-kserve-84bf47c64f-s52pp"
Jul 06 21:02:59.185309 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:02:59.185257    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/241c07fa-93e7-4167-ac56-60eb6807a16b-home\") pod \"sub-test-model-cd7c86-kserve-84bf47c64f-s52pp\" (UID: \"241c07fa-93e7-4167-ac56-60eb6807a16b\") " pod="ai-tenant-e2e-shared-b-cd7c86/sub-test-model-cd7c86-kserve-84bf47c64f-s52pp"
Jul 06 21:02:59.185402 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:02:59.185317    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/241c07fa-93e7-4167-ac56-60eb6807a16b-model-cache\") pod \"sub-test-model-cd7c86-kserve-84bf47c64f-s52pp\" (UID: \"241c07fa-93e7-4167-ac56-60eb6807a16b\") " pod="ai-tenant-e2e-shared-b-cd7c86/sub-test-model-cd7c86-kserve-84bf47c64f-s52pp"
Jul 06 21:02:59.187124 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:02:59.187096    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/241c07fa-93e7-4167-ac56-60eb6807a16b-dshm\") pod \"sub-test-model-cd7c86-kserve-84bf47c64f-s52pp\" (UID: \"241c07fa-93e7-4167-ac56-60eb6807a16b\") " pod="ai-tenant-e2e-shared-b-cd7c86/sub-test-model-cd7c86-kserve-84bf47c64f-s52pp"
Jul 06 21:02:59.187443 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:02:59.187425    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/241c07fa-93e7-4167-ac56-60eb6807a16b-tls-certs\") pod \"sub-test-model-cd7c86-kserve-84bf47c64f-s52pp\" (UID: \"241c07fa-93e7-4167-ac56-60eb6807a16b\") " pod="ai-tenant-e2e-shared-b-cd7c86/sub-test-model-cd7c86-kserve-84bf47c64f-s52pp"
Jul 06 21:02:59.197751 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:02:59.197706    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fdjv\" (UniqueName: \"kubernetes.io/projected/241c07fa-93e7-4167-ac56-60eb6807a16b-kube-api-access-9fdjv\") pod \"sub-test-model-cd7c86-kserve-84bf47c64f-s52pp\" (UID: \"241c07fa-93e7-4167-ac56-60eb6807a16b\") " pod="ai-tenant-e2e-shared-b-cd7c86/sub-test-model-cd7c86-kserve-84bf47c64f-s52pp"
Jul 06 21:02:59.266672 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:02:59.266578    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-cd7c86/sub-test-model-cd7c86-kserve-84bf47c64f-s52pp"
Jul 06 21:02:59.418637 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:02:59.418605    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-cd7c86/sub-test-model-cd7c86-kserve-84bf47c64f-s52pp"]
Jul 06 21:02:59.420169 ip-10-0-141-123 kubenswrapper[2563]: W0706 21:02:59.420142    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod241c07fa_93e7_4167_ac56_60eb6807a16b.slice/crio-cdcc10d219734b5abdc797e7bc2b3a9798503c2b8699e132f60145ce72aac82b WatchSource:0}: Error finding container cdcc10d219734b5abdc797e7bc2b3a9798503c2b8699e132f60145ce72aac82b: Status 404 returned error can't find the container with id cdcc10d219734b5abdc797e7bc2b3a9798503c2b8699e132f60145ce72aac82b
Jul 06 21:02:59.673847 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:02:59.673809    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-cd7c86/sub-test-model-cd7c86-kserve-84bf47c64f-s52pp" event={"ID":"241c07fa-93e7-4167-ac56-60eb6807a16b","Type":"ContainerStarted","Data":"95d5146d3310c8efe8344ec8f0c3ae315e400fdfbdc22dacf635f99862a0e546"}
Jul 06 21:02:59.673847 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:02:59.673851    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-cd7c86/sub-test-model-cd7c86-kserve-84bf47c64f-s52pp" event={"ID":"241c07fa-93e7-4167-ac56-60eb6807a16b","Type":"ContainerStarted","Data":"cdcc10d219734b5abdc797e7bc2b3a9798503c2b8699e132f60145ce72aac82b"}
Jul 06 21:02:59.693881 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:02:59.693830    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-cd7c86/sub-test-model-cd7c86-kserve-84bf47c64f-s52pp" podStartSLOduration=1.693801329 podStartE2EDuration="1.693801329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:02:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:02:59.692840553 +0000 UTC m=+2343.408951214" watchObservedRunningTime="2026-07-06 21:02:59.693801329 +0000 UTC m=+2343.409911989"
Jul 06 21:03:03.508800 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:03:03.508763    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-clwd5"]
Jul 06 21:03:09.266800 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:03:09.266751    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-cd7c86/sub-test-model-cd7c86-kserve-84bf47c64f-s52pp"
Jul 06 21:03:09.267204 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:03:09.266818    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-cd7c86/sub-test-model-cd7c86-kserve-84bf47c64f-s52pp"
Jul 06 21:03:09.279320 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:03:09.279295    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-cd7c86/sub-test-model-cd7c86-kserve-84bf47c64f-s52pp"
Jul 06 21:03:09.732191 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:03:09.732163    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-cd7c86/sub-test-model-cd7c86-kserve-84bf47c64f-s52pp"
Jul 06 21:03:23.700465 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:03:23.700431    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-clwd5"]
Jul 06 21:03:41.564111 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:03:41.564073    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 21:03:42.345723 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:03:42.345689    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-cd7c86/sub-test-model-cd7c86-kserve-84bf47c64f-s52pp"]
Jul 06 21:03:42.377919 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:03:42.377889    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 21:03:45.400513 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:03:45.400474    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-clwd5"]
Jul 06 21:03:56.869320 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:03:56.869282    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-2m7gh_bb2a41aa-a9fc-4ec5-9b78-5e22c1fd301b/console-operator/1.log"
Jul 06 21:03:56.872720 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:03:56.872695    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-2m7gh_bb2a41aa-a9fc-4ec5-9b78-5e22c1fd301b/console-operator/1.log"
Jul 06 21:03:57.365578 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:03:57.365524    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-cd7c86/sub-test-model-cd7c86-kserve-84bf47c64f-s52pp" podUID="241c07fa-93e7-4167-ac56-60eb6807a16b" containerName="main" containerID="cri-o://95d5146d3310c8efe8344ec8f0c3ae315e400fdfbdc22dacf635f99862a0e546" gracePeriod=45
Jul 06 21:03:57.621969 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:03:57.621892    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-cd7c86/sub-test-model-cd7c86-kserve-84bf47c64f-s52pp"
Jul 06 21:03:57.702891 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:03:57.702857    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/241c07fa-93e7-4167-ac56-60eb6807a16b-dshm\") pod \"241c07fa-93e7-4167-ac56-60eb6807a16b\" (UID: \"241c07fa-93e7-4167-ac56-60eb6807a16b\") "
Jul 06 21:03:57.703090 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:03:57.702907    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/241c07fa-93e7-4167-ac56-60eb6807a16b-home\") pod \"241c07fa-93e7-4167-ac56-60eb6807a16b\" (UID: \"241c07fa-93e7-4167-ac56-60eb6807a16b\") "
Jul 06 21:03:57.703090 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:03:57.702960    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/241c07fa-93e7-4167-ac56-60eb6807a16b-tls-certs\") pod \"241c07fa-93e7-4167-ac56-60eb6807a16b\" (UID: \"241c07fa-93e7-4167-ac56-60eb6807a16b\") "
Jul 06 21:03:57.703090 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:03:57.702992    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/241c07fa-93e7-4167-ac56-60eb6807a16b-model-cache\") pod \"241c07fa-93e7-4167-ac56-60eb6807a16b\" (UID: \"241c07fa-93e7-4167-ac56-60eb6807a16b\") "
Jul 06 21:03:57.703090 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:03:57.703037    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9fdjv\" (UniqueName: \"kubernetes.io/projected/241c07fa-93e7-4167-ac56-60eb6807a16b-kube-api-access-9fdjv\") pod \"241c07fa-93e7-4167-ac56-60eb6807a16b\" (UID: \"241c07fa-93e7-4167-ac56-60eb6807a16b\") "
Jul 06 21:03:57.703090 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:03:57.703069    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/241c07fa-93e7-4167-ac56-60eb6807a16b-tmp-dir\") pod \"241c07fa-93e7-4167-ac56-60eb6807a16b\" (UID: \"241c07fa-93e7-4167-ac56-60eb6807a16b\") "
Jul 06 21:03:57.703367 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:03:57.703278    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/241c07fa-93e7-4167-ac56-60eb6807a16b-home" (OuterVolumeSpecName: "home") pod "241c07fa-93e7-4167-ac56-60eb6807a16b" (UID: "241c07fa-93e7-4167-ac56-60eb6807a16b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:03:57.703367 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:03:57.703286    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/241c07fa-93e7-4167-ac56-60eb6807a16b-model-cache" (OuterVolumeSpecName: "model-cache") pod "241c07fa-93e7-4167-ac56-60eb6807a16b" (UID: "241c07fa-93e7-4167-ac56-60eb6807a16b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:03:57.703444 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:03:57.703375    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/241c07fa-93e7-4167-ac56-60eb6807a16b-model-cache\") on node \"ip-10-0-141-123.ec2.internal\" DevicePath \"\""
Jul 06 21:03:57.703444 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:03:57.703385    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/241c07fa-93e7-4167-ac56-60eb6807a16b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "241c07fa-93e7-4167-ac56-60eb6807a16b" (UID: "241c07fa-93e7-4167-ac56-60eb6807a16b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:03:57.703444 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:03:57.703394    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/241c07fa-93e7-4167-ac56-60eb6807a16b-home\") on node \"ip-10-0-141-123.ec2.internal\" DevicePath \"\""
Jul 06 21:03:57.705292 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:03:57.705258    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/241c07fa-93e7-4167-ac56-60eb6807a16b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "241c07fa-93e7-4167-ac56-60eb6807a16b" (UID: "241c07fa-93e7-4167-ac56-60eb6807a16b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 21:03:57.705415 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:03:57.705265    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/241c07fa-93e7-4167-ac56-60eb6807a16b-dshm" (OuterVolumeSpecName: "dshm") pod "241c07fa-93e7-4167-ac56-60eb6807a16b" (UID: "241c07fa-93e7-4167-ac56-60eb6807a16b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:03:57.705415 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:03:57.705332    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/241c07fa-93e7-4167-ac56-60eb6807a16b-kube-api-access-9fdjv" (OuterVolumeSpecName: "kube-api-access-9fdjv") pod "241c07fa-93e7-4167-ac56-60eb6807a16b" (UID: "241c07fa-93e7-4167-ac56-60eb6807a16b"). InnerVolumeSpecName "kube-api-access-9fdjv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 21:03:57.804422 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:03:57.804377    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9fdjv\" (UniqueName: \"kubernetes.io/projected/241c07fa-93e7-4167-ac56-60eb6807a16b-kube-api-access-9fdjv\") on node \"ip-10-0-141-123.ec2.internal\" DevicePath \"\""
Jul 06 21:03:57.804422 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:03:57.804412    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/241c07fa-93e7-4167-ac56-60eb6807a16b-tmp-dir\") on node \"ip-10-0-141-123.ec2.internal\" DevicePath \"\""
Jul 06 21:03:57.804422 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:03:57.804426    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/241c07fa-93e7-4167-ac56-60eb6807a16b-dshm\") on node \"ip-10-0-141-123.ec2.internal\" DevicePath \"\""
Jul 06 21:03:57.804697 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:03:57.804436    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/241c07fa-93e7-4167-ac56-60eb6807a16b-tls-certs\") on node \"ip-10-0-141-123.ec2.internal\" DevicePath \"\""
Jul 06 21:03:57.958660 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:03:57.958566    2563 generic.go:356] "Generic (PLEG): container finished" podID="241c07fa-93e7-4167-ac56-60eb6807a16b" containerID="95d5146d3310c8efe8344ec8f0c3ae315e400fdfbdc22dacf635f99862a0e546" exitCode=0
Jul 06 21:03:57.958660 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:03:57.958624    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-cd7c86/sub-test-model-cd7c86-kserve-84bf47c64f-s52pp"
Jul 06 21:03:57.958660 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:03:57.958633    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-cd7c86/sub-test-model-cd7c86-kserve-84bf47c64f-s52pp" event={"ID":"241c07fa-93e7-4167-ac56-60eb6807a16b","Type":"ContainerDied","Data":"95d5146d3310c8efe8344ec8f0c3ae315e400fdfbdc22dacf635f99862a0e546"}
Jul 06 21:03:57.959257 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:03:57.958673    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-cd7c86/sub-test-model-cd7c86-kserve-84bf47c64f-s52pp" event={"ID":"241c07fa-93e7-4167-ac56-60eb6807a16b","Type":"ContainerDied","Data":"cdcc10d219734b5abdc797e7bc2b3a9798503c2b8699e132f60145ce72aac82b"}
Jul 06 21:03:57.959257 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:03:57.958695    2563 scope.go:117] "RemoveContainer" containerID="95d5146d3310c8efe8344ec8f0c3ae315e400fdfbdc22dacf635f99862a0e546"
Jul 06 21:03:57.967688 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:03:57.967666    2563 scope.go:117] "RemoveContainer" containerID="95d5146d3310c8efe8344ec8f0c3ae315e400fdfbdc22dacf635f99862a0e546"
Jul 06 21:03:57.968001 ip-10-0-141-123 kubenswrapper[2563]: E0706 21:03:57.967981    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95d5146d3310c8efe8344ec8f0c3ae315e400fdfbdc22dacf635f99862a0e546\": container with ID starting with 95d5146d3310c8efe8344ec8f0c3ae315e400fdfbdc22dacf635f99862a0e546 not found: ID does not exist" containerID="95d5146d3310c8efe8344ec8f0c3ae315e400fdfbdc22dacf635f99862a0e546"
Jul 06 21:03:57.968075 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:03:57.968012    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95d5146d3310c8efe8344ec8f0c3ae315e400fdfbdc22dacf635f99862a0e546"} err="failed to get container status \"95d5146d3310c8efe8344ec8f0c3ae315e400fdfbdc22dacf635f99862a0e546\": rpc error: code = NotFound desc = could not find container \"95d5146d3310c8efe8344ec8f0c3ae315e400fdfbdc22dacf635f99862a0e546\": container with ID starting with 95d5146d3310c8efe8344ec8f0c3ae315e400fdfbdc22dacf635f99862a0e546 not found: ID does not exist"
Jul 06 21:03:57.990571 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:03:57.990539    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-cd7c86/sub-test-model-cd7c86-kserve-84bf47c64f-s52pp"]
Jul 06 21:03:57.999116 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:03:57.999084    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-cd7c86/sub-test-model-cd7c86-kserve-84bf47c64f-s52pp"]
Jul 06 21:03:58.834218 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:03:58.834183    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="241c07fa-93e7-4167-ac56-60eb6807a16b" path="/var/lib/kubelet/pods/241c07fa-93e7-4167-ac56-60eb6807a16b/volumes"
Jul 06 21:04:03.867027 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:04:03.866988    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-544d30/rate-test-model-544d30-kserve-5455cf8899-j7pbf"]
Jul 06 21:04:03.867515 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:04:03.867492    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="241c07fa-93e7-4167-ac56-60eb6807a16b" containerName="main"
Jul 06 21:04:03.867515 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:04:03.867510    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="241c07fa-93e7-4167-ac56-60eb6807a16b" containerName="main"
Jul 06 21:04:03.867654 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:04:03.867639    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="241c07fa-93e7-4167-ac56-60eb6807a16b" containerName="main"
Jul 06 21:04:03.870668 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:04:03.870648    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-544d30/rate-test-model-544d30-kserve-5455cf8899-j7pbf"
Jul 06 21:04:03.874138 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:04:03.874121    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-544d30\"/\"rate-test-model-544d30-kserve-self-signed-certs\""
Jul 06 21:04:03.874251 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:04:03.874123    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-544d30\"/\"default-dockercfg-pjfp2\""
Jul 06 21:04:03.874251 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:04:03.874124    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-544d30\"/\"kube-root-ca.crt\""
Jul 06 21:04:03.877513 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:04:03.877497    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-544d30\"/\"openshift-service-ca.crt\""
Jul 06 21:04:03.887997 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:04:03.887972    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-544d30/rate-test-model-544d30-kserve-5455cf8899-j7pbf"]
Jul 06 21:04:03.938386 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:04:03.938350    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 21:04:03.960606 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:04:03.960571    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2d12373-de22-42a1-8871-2dd766c6b83e-model-cache\") pod \"rate-test-model-544d30-kserve-5455cf8899-j7pbf\" (UID: \"e2d12373-de22-42a1-8871-2dd766c6b83e\") " pod="ai-tenant-e2e-shared-a-544d30/rate-test-model-544d30-kserve-5455cf8899-j7pbf"
Jul 06 21:04:03.960810 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:04:03.960616    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2d12373-de22-42a1-8871-2dd766c6b83e-tmp-dir\") pod \"rate-test-model-544d30-kserve-5455cf8899-j7pbf\" (UID: \"e2d12373-de22-42a1-8871-2dd766c6b83e\") " pod="ai-tenant-e2e-shared-a-544d30/rate-test-model-544d30-kserve-5455cf8899-j7pbf"
Jul 06 21:04:03.960810 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:04:03.960686    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4sf7\" (UniqueName: \"kubernetes.io/projected/e2d12373-de22-42a1-8871-2dd766c6b83e-kube-api-access-t4sf7\") pod \"rate-test-model-544d30-kserve-5455cf8899-j7pbf\" (UID: \"e2d12373-de22-42a1-8871-2dd766c6b83e\") " pod="ai-tenant-e2e-shared-a-544d30/rate-test-model-544d30-kserve-5455cf8899-j7pbf"
Jul 06 21:04:03.960810 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:04:03.960780    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2d12373-de22-42a1-8871-2dd766c6b83e-home\") pod \"rate-test-model-544d30-kserve-5455cf8899-j7pbf\" (UID: \"e2d12373-de22-42a1-8871-2dd766c6b83e\") " pod="ai-tenant-e2e-shared-a-544d30/rate-test-model-544d30-kserve-5455cf8899-j7pbf"
Jul 06 21:04:03.960810 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:04:03.960803    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2d12373-de22-42a1-8871-2dd766c6b83e-dshm\") pod \"rate-test-model-544d30-kserve-5455cf8899-j7pbf\" (UID: \"e2d12373-de22-42a1-8871-2dd766c6b83e\") " pod="ai-tenant-e2e-shared-a-544d30/rate-test-model-544d30-kserve-5455cf8899-j7pbf"
Jul 06 21:04:03.961008 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:04:03.960880    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2d12373-de22-42a1-8871-2dd766c6b83e-tls-certs\") pod \"rate-test-model-544d30-kserve-5455cf8899-j7pbf\" (UID: \"e2d12373-de22-42a1-8871-2dd766c6b83e\") " pod="ai-tenant-e2e-shared-a-544d30/rate-test-model-544d30-kserve-5455cf8899-j7pbf"
Jul 06 21:04:04.062059 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:04:04.062024    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2d12373-de22-42a1-8871-2dd766c6b83e-tls-certs\") pod \"rate-test-model-544d30-kserve-5455cf8899-j7pbf\" (UID: \"e2d12373-de22-42a1-8871-2dd766c6b83e\") " pod="ai-tenant-e2e-shared-a-544d30/rate-test-model-544d30-kserve-5455cf8899-j7pbf"
Jul 06 21:04:04.062260 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:04:04.062089    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2d12373-de22-42a1-8871-2dd766c6b83e-model-cache\") pod \"rate-test-model-544d30-kserve-5455cf8899-j7pbf\" (UID: \"e2d12373-de22-42a1-8871-2dd766c6b83e\") " pod="ai-tenant-e2e-shared-a-544d30/rate-test-model-544d30-kserve-5455cf8899-j7pbf"
Jul 06 21:04:04.062260 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:04:04.062123    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2d12373-de22-42a1-8871-2dd766c6b83e-tmp-dir\") pod \"rate-test-model-544d30-kserve-5455cf8899-j7pbf\" (UID: \"e2d12373-de22-42a1-8871-2dd766c6b83e\") " pod="ai-tenant-e2e-shared-a-544d30/rate-test-model-544d30-kserve-5455cf8899-j7pbf"
Jul 06 21:04:04.062260 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:04:04.062161    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4sf7\" (UniqueName: \"kubernetes.io/projected/e2d12373-de22-42a1-8871-2dd766c6b83e-kube-api-access-t4sf7\") pod \"rate-test-model-544d30-kserve-5455cf8899-j7pbf\" (UID: \"e2d12373-de22-42a1-8871-2dd766c6b83e\") " pod="ai-tenant-e2e-shared-a-544d30/rate-test-model-544d30-kserve-5455cf8899-j7pbf"
Jul 06 21:04:04.062260 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:04:04.062220    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2d12373-de22-42a1-8871-2dd766c6b83e-home\") pod \"rate-test-model-544d30-kserve-5455cf8899-j7pbf\" (UID: \"e2d12373-de22-42a1-8871-2dd766c6b83e\") " pod="ai-tenant-e2e-shared-a-544d30/rate-test-model-544d30-kserve-5455cf8899-j7pbf"
Jul 06 21:04:04.062457 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:04:04.062242    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2d12373-de22-42a1-8871-2dd766c6b83e-dshm\") pod \"rate-test-model-544d30-kserve-5455cf8899-j7pbf\" (UID: \"e2d12373-de22-42a1-8871-2dd766c6b83e\") " pod="ai-tenant-e2e-shared-a-544d30/rate-test-model-544d30-kserve-5455cf8899-j7pbf"
Jul 06 21:04:04.062597 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:04:04.062563    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2d12373-de22-42a1-8871-2dd766c6b83e-model-cache\") pod \"rate-test-model-544d30-kserve-5455cf8899-j7pbf\" (UID: \"e2d12373-de22-42a1-8871-2dd766c6b83e\") " pod="ai-tenant-e2e-shared-a-544d30/rate-test-model-544d30-kserve-5455cf8899-j7pbf"
Jul 06 21:04:04.062597 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:04:04.062569    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2d12373-de22-42a1-8871-2dd766c6b83e-tmp-dir\") pod \"rate-test-model-544d30-kserve-5455cf8899-j7pbf\" (UID: \"e2d12373-de22-42a1-8871-2dd766c6b83e\") " pod="ai-tenant-e2e-shared-a-544d30/rate-test-model-544d30-kserve-5455cf8899-j7pbf"
Jul 06 21:04:04.062794 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:04:04.062633    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2d12373-de22-42a1-8871-2dd766c6b83e-home\") pod \"rate-test-model-544d30-kserve-5455cf8899-j7pbf\" (UID: \"e2d12373-de22-42a1-8871-2dd766c6b83e\") " pod="ai-tenant-e2e-shared-a-544d30/rate-test-model-544d30-kserve-5455cf8899-j7pbf"
Jul 06 21:04:04.064375 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:04:04.064354    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2d12373-de22-42a1-8871-2dd766c6b83e-dshm\") pod \"rate-test-model-544d30-kserve-5455cf8899-j7pbf\" (UID: \"e2d12373-de22-42a1-8871-2dd766c6b83e\") " pod="ai-tenant-e2e-shared-a-544d30/rate-test-model-544d30-kserve-5455cf8899-j7pbf"
Jul 06 21:04:04.064544 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:04:04.064527    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2d12373-de22-42a1-8871-2dd766c6b83e-tls-certs\") pod \"rate-test-model-544d30-kserve-5455cf8899-j7pbf\" (UID: \"e2d12373-de22-42a1-8871-2dd766c6b83e\") " pod="ai-tenant-e2e-shared-a-544d30/rate-test-model-544d30-kserve-5455cf8899-j7pbf"
Jul 06 21:04:04.073057 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:04:04.073036    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4sf7\" (UniqueName: \"kubernetes.io/projected/e2d12373-de22-42a1-8871-2dd766c6b83e-kube-api-access-t4sf7\") pod \"rate-test-model-544d30-kserve-5455cf8899-j7pbf\" (UID: \"e2d12373-de22-42a1-8871-2dd766c6b83e\") " pod="ai-tenant-e2e-shared-a-544d30/rate-test-model-544d30-kserve-5455cf8899-j7pbf"
Jul 06 21:04:04.180210 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:04:04.180111    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-544d30/rate-test-model-544d30-kserve-5455cf8899-j7pbf"
Jul 06 21:04:04.328905 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:04:04.328841    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-544d30/rate-test-model-544d30-kserve-5455cf8899-j7pbf"]
Jul 06 21:04:04.331222 ip-10-0-141-123 kubenswrapper[2563]: W0706 21:04:04.331195    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode2d12373_de22_42a1_8871_2dd766c6b83e.slice/crio-b3af6536ecda98ad32f947f21207dc35094c827878255e803d9ad6146661b9c6 WatchSource:0}: Error finding container b3af6536ecda98ad32f947f21207dc35094c827878255e803d9ad6146661b9c6: Status 404 returned error can't find the container with id b3af6536ecda98ad32f947f21207dc35094c827878255e803d9ad6146661b9c6
Jul 06 21:04:04.993991 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:04:04.993838    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-544d30/rate-test-model-544d30-kserve-5455cf8899-j7pbf" event={"ID":"e2d12373-de22-42a1-8871-2dd766c6b83e","Type":"ContainerStarted","Data":"0392a39a7fee7bc35b0b76d48cc4914b846b9d3ff1d98089c0c428c43ac9c55c"}
Jul 06 21:04:04.993991 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:04:04.993890    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-544d30/rate-test-model-544d30-kserve-5455cf8899-j7pbf" event={"ID":"e2d12373-de22-42a1-8871-2dd766c6b83e","Type":"ContainerStarted","Data":"b3af6536ecda98ad32f947f21207dc35094c827878255e803d9ad6146661b9c6"}
Jul 06 21:04:05.075139 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:04:05.075087    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-544d30/rate-test-model-544d30-kserve-5455cf8899-j7pbf" podStartSLOduration=2.075070333 podStartE2EDuration="2.075070333s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:04:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:04:05.074550399 +0000 UTC m=+2408.790661060" watchObservedRunningTime="2026-07-06 21:04:05.075070333 +0000 UTC m=+2408.791180994"
Jul 06 21:04:07.628109 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:04:07.628078    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-clwd5"]
Jul 06 21:04:12.148240 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:04:12.148200    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 21:04:12.289639 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:04:12.289607    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 21:04:14.180359 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:04:14.180319    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-544d30/rate-test-model-544d30-kserve-5455cf8899-j7pbf"
Jul 06 21:04:14.180806 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:04:14.180391    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-544d30/rate-test-model-544d30-kserve-5455cf8899-j7pbf"
Jul 06 21:04:14.193130 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:04:14.193109    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-544d30/rate-test-model-544d30-kserve-5455cf8899-j7pbf"
Jul 06 21:04:15.051894 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:04:15.051865    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-544d30/rate-test-model-544d30-kserve-5455cf8899-j7pbf"
Jul 06 21:04:37.381022 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:04:37.380988    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-cd7c86/rate-test-model-cd7c86-kserve-6c9d768687-xgd2j"]
Jul 06 21:04:37.384266 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:04:37.384250    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-cd7c86/rate-test-model-cd7c86-kserve-6c9d768687-xgd2j"
Jul 06 21:04:37.391853 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:04:37.391832    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-cd7c86\"/\"default-dockercfg-wqfc5\""
Jul 06 21:04:37.391953 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:04:37.391852    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-cd7c86\"/\"rate-test-model-cd7c86-kserve-self-signed-certs\""
Jul 06 21:04:37.392572 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:04:37.392548    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-cd7c86\"/\"openshift-service-ca.crt\""
Jul 06 21:04:37.395756 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:04:37.395738    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-cd7c86\"/\"kube-root-ca.crt\""
Jul 06 21:04:37.414008 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:04:37.413972    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-cd7c86/rate-test-model-cd7c86-kserve-6c9d768687-xgd2j"]
Jul 06 21:04:37.422884 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:04:37.422851    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 21:04:37.478707 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:04:37.478674    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80888cd0-0963-4670-99d6-f33332af02f6-tmp-dir\") pod \"rate-test-model-cd7c86-kserve-6c9d768687-xgd2j\" (UID: \"80888cd0-0963-4670-99d6-f33332af02f6\") " pod="ai-tenant-e2e-shared-b-cd7c86/rate-test-model-cd7c86-kserve-6c9d768687-xgd2j"
Jul 06 21:04:37.478930 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:04:37.478717    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80888cd0-0963-4670-99d6-f33332af02f6-home\") pod \"rate-test-model-cd7c86-kserve-6c9d768687-xgd2j\" (UID: \"80888cd0-0963-4670-99d6-f33332af02f6\") " pod="ai-tenant-e2e-shared-b-cd7c86/rate-test-model-cd7c86-kserve-6c9d768687-xgd2j"
Jul 06 21:04:37.478930 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:04:37.478842    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80888cd0-0963-4670-99d6-f33332af02f6-tls-certs\") pod \"rate-test-model-cd7c86-kserve-6c9d768687-xgd2j\" (UID: \"80888cd0-0963-4670-99d6-f33332af02f6\") " pod="ai-tenant-e2e-shared-b-cd7c86/rate-test-model-cd7c86-kserve-6c9d768687-xgd2j"
Jul 06 21:04:37.478930 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:04:37.478896    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80888cd0-0963-4670-99d6-f33332af02f6-model-cache\") pod \"rate-test-model-cd7c86-kserve-6c9d768687-xgd2j\" (UID: \"80888cd0-0963-4670-99d6-f33332af02f6\") " pod="ai-tenant-e2e-shared-b-cd7c86/rate-test-model-cd7c86-kserve-6c9d768687-xgd2j"
Jul 06 21:04:37.479060 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:04:37.478966    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80888cd0-0963-4670-99d6-f33332af02f6-dshm\") pod \"rate-test-model-cd7c86-kserve-6c9d768687-xgd2j\" (UID: \"80888cd0-0963-4670-99d6-f33332af02f6\") " pod="ai-tenant-e2e-shared-b-cd7c86/rate-test-model-cd7c86-kserve-6c9d768687-xgd2j"
Jul 06 21:04:37.479060 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:04:37.479041    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-schwp\" (UniqueName: \"kubernetes.io/projected/80888cd0-0963-4670-99d6-f33332af02f6-kube-api-access-schwp\") pod \"rate-test-model-cd7c86-kserve-6c9d768687-xgd2j\" (UID: \"80888cd0-0963-4670-99d6-f33332af02f6\") " pod="ai-tenant-e2e-shared-b-cd7c86/rate-test-model-cd7c86-kserve-6c9d768687-xgd2j"
Jul 06 21:04:37.579536 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:04:37.579500    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80888cd0-0963-4670-99d6-f33332af02f6-tls-certs\") pod \"rate-test-model-cd7c86-kserve-6c9d768687-xgd2j\" (UID: \"80888cd0-0963-4670-99d6-f33332af02f6\") " pod="ai-tenant-e2e-shared-b-cd7c86/rate-test-model-cd7c86-kserve-6c9d768687-xgd2j"
Jul 06 21:04:37.579771 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:04:37.579556    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80888cd0-0963-4670-99d6-f33332af02f6-model-cache\") pod \"rate-test-model-cd7c86-kserve-6c9d768687-xgd2j\" (UID: \"80888cd0-0963-4670-99d6-f33332af02f6\") " pod="ai-tenant-e2e-shared-b-cd7c86/rate-test-model-cd7c86-kserve-6c9d768687-xgd2j"
Jul 06 21:04:37.579771 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:04:37.579595    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80888cd0-0963-4670-99d6-f33332af02f6-dshm\") pod \"rate-test-model-cd7c86-kserve-6c9d768687-xgd2j\" (UID: \"80888cd0-0963-4670-99d6-f33332af02f6\") " pod="ai-tenant-e2e-shared-b-cd7c86/rate-test-model-cd7c86-kserve-6c9d768687-xgd2j"
Jul 06 21:04:37.579771 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:04:37.579649    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-schwp\" (UniqueName: \"kubernetes.io/projected/80888cd0-0963-4670-99d6-f33332af02f6-kube-api-access-schwp\") pod \"rate-test-model-cd7c86-kserve-6c9d768687-xgd2j\" (UID: \"80888cd0-0963-4670-99d6-f33332af02f6\") " pod="ai-tenant-e2e-shared-b-cd7c86/rate-test-model-cd7c86-kserve-6c9d768687-xgd2j"
Jul 06 21:04:37.579771 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:04:37.579686    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80888cd0-0963-4670-99d6-f33332af02f6-tmp-dir\") pod \"rate-test-model-cd7c86-kserve-6c9d768687-xgd2j\" (UID: \"80888cd0-0963-4670-99d6-f33332af02f6\") " pod="ai-tenant-e2e-shared-b-cd7c86/rate-test-model-cd7c86-kserve-6c9d768687-xgd2j"
Jul 06 21:04:37.579771 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:04:37.579714    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80888cd0-0963-4670-99d6-f33332af02f6-home\") pod \"rate-test-model-cd7c86-kserve-6c9d768687-xgd2j\" (UID: \"80888cd0-0963-4670-99d6-f33332af02f6\") " pod="ai-tenant-e2e-shared-b-cd7c86/rate-test-model-cd7c86-kserve-6c9d768687-xgd2j"
Jul 06 21:04:37.580117 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:04:37.580093    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80888cd0-0963-4670-99d6-f33332af02f6-model-cache\") pod \"rate-test-model-cd7c86-kserve-6c9d768687-xgd2j\" (UID: \"80888cd0-0963-4670-99d6-f33332af02f6\") " pod="ai-tenant-e2e-shared-b-cd7c86/rate-test-model-cd7c86-kserve-6c9d768687-xgd2j"
Jul 06 21:04:37.580188 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:04:37.580106    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80888cd0-0963-4670-99d6-f33332af02f6-home\") pod \"rate-test-model-cd7c86-kserve-6c9d768687-xgd2j\" (UID: \"80888cd0-0963-4670-99d6-f33332af02f6\") " pod="ai-tenant-e2e-shared-b-cd7c86/rate-test-model-cd7c86-kserve-6c9d768687-xgd2j"
Jul 06 21:04:37.580188 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:04:37.580177    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80888cd0-0963-4670-99d6-f33332af02f6-tmp-dir\") pod \"rate-test-model-cd7c86-kserve-6c9d768687-xgd2j\" (UID: \"80888cd0-0963-4670-99d6-f33332af02f6\") " pod="ai-tenant-e2e-shared-b-cd7c86/rate-test-model-cd7c86-kserve-6c9d768687-xgd2j"
Jul 06 21:04:37.581963 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:04:37.581935    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80888cd0-0963-4670-99d6-f33332af02f6-dshm\") pod \"rate-test-model-cd7c86-kserve-6c9d768687-xgd2j\" (UID: \"80888cd0-0963-4670-99d6-f33332af02f6\") " pod="ai-tenant-e2e-shared-b-cd7c86/rate-test-model-cd7c86-kserve-6c9d768687-xgd2j"
Jul 06 21:04:37.582224 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:04:37.582207    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80888cd0-0963-4670-99d6-f33332af02f6-tls-certs\") pod \"rate-test-model-cd7c86-kserve-6c9d768687-xgd2j\" (UID: \"80888cd0-0963-4670-99d6-f33332af02f6\") " pod="ai-tenant-e2e-shared-b-cd7c86/rate-test-model-cd7c86-kserve-6c9d768687-xgd2j"
Jul 06 21:04:37.592272 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:04:37.592249    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-schwp\" (UniqueName: \"kubernetes.io/projected/80888cd0-0963-4670-99d6-f33332af02f6-kube-api-access-schwp\") pod \"rate-test-model-cd7c86-kserve-6c9d768687-xgd2j\" (UID: \"80888cd0-0963-4670-99d6-f33332af02f6\") " pod="ai-tenant-e2e-shared-b-cd7c86/rate-test-model-cd7c86-kserve-6c9d768687-xgd2j"
Jul 06 21:04:37.694435 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:04:37.694334    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-cd7c86/rate-test-model-cd7c86-kserve-6c9d768687-xgd2j"
Jul 06 21:04:37.856576 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:04:37.856553    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-cd7c86/rate-test-model-cd7c86-kserve-6c9d768687-xgd2j"]
Jul 06 21:04:37.859207 ip-10-0-141-123 kubenswrapper[2563]: W0706 21:04:37.859184    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod80888cd0_0963_4670_99d6_f33332af02f6.slice/crio-f573cbee67dcbdda744418d84e650792db4ac40911542f5a8a8da75dd2ec1c55 WatchSource:0}: Error finding container f573cbee67dcbdda744418d84e650792db4ac40911542f5a8a8da75dd2ec1c55: Status 404 returned error can't find the container with id f573cbee67dcbdda744418d84e650792db4ac40911542f5a8a8da75dd2ec1c55
Jul 06 21:04:38.149409 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:04:38.149377    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-cd7c86/rate-test-model-cd7c86-kserve-6c9d768687-xgd2j" event={"ID":"80888cd0-0963-4670-99d6-f33332af02f6","Type":"ContainerStarted","Data":"7d84dcfe250b0fa859025e984bc0e5014a0d8c7bd0f876481475182d6e86abec"}
Jul 06 21:04:38.149409 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:04:38.149414    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-cd7c86/rate-test-model-cd7c86-kserve-6c9d768687-xgd2j" event={"ID":"80888cd0-0963-4670-99d6-f33332af02f6","Type":"ContainerStarted","Data":"f573cbee67dcbdda744418d84e650792db4ac40911542f5a8a8da75dd2ec1c55"}
Jul 06 21:04:38.173619 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:04:38.173554    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-cd7c86/rate-test-model-cd7c86-kserve-6c9d768687-xgd2j" podStartSLOduration=1.173537192 podStartE2EDuration="1.173537192s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:04:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:04:38.171037226 +0000 UTC m=+2441.887147889" watchObservedRunningTime="2026-07-06 21:04:38.173537192 +0000 UTC m=+2441.889647852"
Jul 06 21:04:41.640134 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:04:41.640097    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-clwd5"]
Jul 06 21:04:47.695381 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:04:47.695346    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-cd7c86/rate-test-model-cd7c86-kserve-6c9d768687-xgd2j"
Jul 06 21:04:47.695815 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:04:47.695413    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-cd7c86/rate-test-model-cd7c86-kserve-6c9d768687-xgd2j"
Jul 06 21:04:47.708136 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:04:47.708104    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-cd7c86/rate-test-model-cd7c86-kserve-6c9d768687-xgd2j"
Jul 06 21:04:48.209507 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:04:48.209476    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-cd7c86/rate-test-model-cd7c86-kserve-6c9d768687-xgd2j"
Jul 06 21:04:55.503088 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:04:55.503054    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-clwd5"]
Jul 06 21:05:02.171094 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:05:02.171062    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 21:05:02.271995 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:05:02.271962    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 21:05:11.524456 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:05:11.524420    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 21:05:11.628373 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:05:11.628338    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 21:05:13.797006 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:05:13.796961    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-clwd5"]
Jul 06 21:05:17.913885 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:05:17.913848    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-544d30/rate-test-model-544d30-kserve-5455cf8899-j7pbf"]
Jul 06 21:05:17.945654 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:05:17.945623    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 21:05:18.903287 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:05:18.903251    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-cd7c86/rate-test-model-cd7c86-kserve-6c9d768687-xgd2j"]
Jul 06 21:05:18.919176 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:05:18.919147    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 21:05:23.660133 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:05:23.660099    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 21:05:24.122336 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:05:24.122305    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 21:05:24.399076 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:05:24.398991    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 21:05:24.507111 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:05:24.507076    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 21:05:24.747993 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:05:24.747916    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 21:05:29.171118 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:05:29.171087    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 21:05:29.198538 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:05:29.198498    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-clwd5"]
Jul 06 21:05:32.933452 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:05:32.933411    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-544d30/rate-test-model-544d30-kserve-5455cf8899-j7pbf" podUID="e2d12373-de22-42a1-8871-2dd766c6b83e" containerName="main" containerID="cri-o://0392a39a7fee7bc35b0b76d48cc4914b846b9d3ff1d98089c0c428c43ac9c55c" gracePeriod=45
Jul 06 21:05:33.174986 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:05:33.174958    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-544d30/rate-test-model-544d30-kserve-5455cf8899-j7pbf"
Jul 06 21:05:33.301492 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:05:33.301389    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t4sf7\" (UniqueName: \"kubernetes.io/projected/e2d12373-de22-42a1-8871-2dd766c6b83e-kube-api-access-t4sf7\") pod \"e2d12373-de22-42a1-8871-2dd766c6b83e\" (UID: \"e2d12373-de22-42a1-8871-2dd766c6b83e\") "
Jul 06 21:05:33.301492 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:05:33.301440    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2d12373-de22-42a1-8871-2dd766c6b83e-home\") pod \"e2d12373-de22-42a1-8871-2dd766c6b83e\" (UID: \"e2d12373-de22-42a1-8871-2dd766c6b83e\") "
Jul 06 21:05:33.301492 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:05:33.301480    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2d12373-de22-42a1-8871-2dd766c6b83e-tls-certs\") pod \"e2d12373-de22-42a1-8871-2dd766c6b83e\" (UID: \"e2d12373-de22-42a1-8871-2dd766c6b83e\") "
Jul 06 21:05:33.301870 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:05:33.301512    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2d12373-de22-42a1-8871-2dd766c6b83e-dshm\") pod \"e2d12373-de22-42a1-8871-2dd766c6b83e\" (UID: \"e2d12373-de22-42a1-8871-2dd766c6b83e\") "
Jul 06 21:05:33.301870 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:05:33.301555    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2d12373-de22-42a1-8871-2dd766c6b83e-tmp-dir\") pod \"e2d12373-de22-42a1-8871-2dd766c6b83e\" (UID: \"e2d12373-de22-42a1-8871-2dd766c6b83e\") "
Jul 06 21:05:33.301870 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:05:33.301574    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2d12373-de22-42a1-8871-2dd766c6b83e-model-cache\") pod \"e2d12373-de22-42a1-8871-2dd766c6b83e\" (UID: \"e2d12373-de22-42a1-8871-2dd766c6b83e\") "
Jul 06 21:05:33.301870 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:05:33.301804    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2d12373-de22-42a1-8871-2dd766c6b83e-home" (OuterVolumeSpecName: "home") pod "e2d12373-de22-42a1-8871-2dd766c6b83e" (UID: "e2d12373-de22-42a1-8871-2dd766c6b83e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:05:33.302093 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:05:33.302024    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2d12373-de22-42a1-8871-2dd766c6b83e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e2d12373-de22-42a1-8871-2dd766c6b83e" (UID: "e2d12373-de22-42a1-8871-2dd766c6b83e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:05:33.302144 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:05:33.302119    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2d12373-de22-42a1-8871-2dd766c6b83e-model-cache" (OuterVolumeSpecName: "model-cache") pod "e2d12373-de22-42a1-8871-2dd766c6b83e" (UID: "e2d12373-de22-42a1-8871-2dd766c6b83e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:05:33.303654 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:05:33.303626    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2d12373-de22-42a1-8871-2dd766c6b83e-dshm" (OuterVolumeSpecName: "dshm") pod "e2d12373-de22-42a1-8871-2dd766c6b83e" (UID: "e2d12373-de22-42a1-8871-2dd766c6b83e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:05:33.303783 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:05:33.303692    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2d12373-de22-42a1-8871-2dd766c6b83e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e2d12373-de22-42a1-8871-2dd766c6b83e" (UID: "e2d12373-de22-42a1-8871-2dd766c6b83e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 21:05:33.303783 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:05:33.303694    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2d12373-de22-42a1-8871-2dd766c6b83e-kube-api-access-t4sf7" (OuterVolumeSpecName: "kube-api-access-t4sf7") pod "e2d12373-de22-42a1-8871-2dd766c6b83e" (UID: "e2d12373-de22-42a1-8871-2dd766c6b83e"). InnerVolumeSpecName "kube-api-access-t4sf7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 21:05:33.402324 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:05:33.402277    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4sf7\" (UniqueName: \"kubernetes.io/projected/e2d12373-de22-42a1-8871-2dd766c6b83e-kube-api-access-t4sf7\") on node \"ip-10-0-141-123.ec2.internal\" DevicePath \"\""
Jul 06 21:05:33.402324 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:05:33.402320    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2d12373-de22-42a1-8871-2dd766c6b83e-home\") on node \"ip-10-0-141-123.ec2.internal\" DevicePath \"\""
Jul 06 21:05:33.402324 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:05:33.402332    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2d12373-de22-42a1-8871-2dd766c6b83e-tls-certs\") on node \"ip-10-0-141-123.ec2.internal\" DevicePath \"\""
Jul 06 21:05:33.402324 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:05:33.402342    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2d12373-de22-42a1-8871-2dd766c6b83e-dshm\") on node \"ip-10-0-141-123.ec2.internal\" DevicePath \"\""
Jul 06 21:05:33.402604 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:05:33.402349    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2d12373-de22-42a1-8871-2dd766c6b83e-tmp-dir\") on node \"ip-10-0-141-123.ec2.internal\" DevicePath \"\""
Jul 06 21:05:33.402604 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:05:33.402356    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2d12373-de22-42a1-8871-2dd766c6b83e-model-cache\") on node \"ip-10-0-141-123.ec2.internal\" DevicePath \"\""
Jul 06 21:05:33.416486 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:05:33.416454    2563 generic.go:356] "Generic (PLEG): container finished" podID="e2d12373-de22-42a1-8871-2dd766c6b83e" containerID="0392a39a7fee7bc35b0b76d48cc4914b846b9d3ff1d98089c0c428c43ac9c55c" exitCode=0
Jul 06 21:05:33.416672 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:05:33.416496    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-544d30/rate-test-model-544d30-kserve-5455cf8899-j7pbf"
Jul 06 21:05:33.416672 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:05:33.416541    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-544d30/rate-test-model-544d30-kserve-5455cf8899-j7pbf" event={"ID":"e2d12373-de22-42a1-8871-2dd766c6b83e","Type":"ContainerDied","Data":"0392a39a7fee7bc35b0b76d48cc4914b846b9d3ff1d98089c0c428c43ac9c55c"}
Jul 06 21:05:33.416672 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:05:33.416589    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-544d30/rate-test-model-544d30-kserve-5455cf8899-j7pbf" event={"ID":"e2d12373-de22-42a1-8871-2dd766c6b83e","Type":"ContainerDied","Data":"b3af6536ecda98ad32f947f21207dc35094c827878255e803d9ad6146661b9c6"}
Jul 06 21:05:33.416672 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:05:33.416615    2563 scope.go:117] "RemoveContainer" containerID="0392a39a7fee7bc35b0b76d48cc4914b846b9d3ff1d98089c0c428c43ac9c55c"
Jul 06 21:05:33.425786 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:05:33.425769    2563 scope.go:117] "RemoveContainer" containerID="0392a39a7fee7bc35b0b76d48cc4914b846b9d3ff1d98089c0c428c43ac9c55c"
Jul 06 21:05:33.426077 ip-10-0-141-123 kubenswrapper[2563]: E0706 21:05:33.426056    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0392a39a7fee7bc35b0b76d48cc4914b846b9d3ff1d98089c0c428c43ac9c55c\": container with ID starting with 0392a39a7fee7bc35b0b76d48cc4914b846b9d3ff1d98089c0c428c43ac9c55c not found: ID does not exist" containerID="0392a39a7fee7bc35b0b76d48cc4914b846b9d3ff1d98089c0c428c43ac9c55c"
Jul 06 21:05:33.426140 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:05:33.426084    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0392a39a7fee7bc35b0b76d48cc4914b846b9d3ff1d98089c0c428c43ac9c55c"} err="failed to get container status \"0392a39a7fee7bc35b0b76d48cc4914b846b9d3ff1d98089c0c428c43ac9c55c\": rpc error: code = NotFound desc = could not find container \"0392a39a7fee7bc35b0b76d48cc4914b846b9d3ff1d98089c0c428c43ac9c55c\": container with ID starting with 0392a39a7fee7bc35b0b76d48cc4914b846b9d3ff1d98089c0c428c43ac9c55c not found: ID does not exist"
Jul 06 21:05:33.440322 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:05:33.440292    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-544d30/rate-test-model-544d30-kserve-5455cf8899-j7pbf"]
Jul 06 21:05:33.443316 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:05:33.443292    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-544d30/rate-test-model-544d30-kserve-5455cf8899-j7pbf"]
Jul 06 21:05:33.923486 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:05:33.923440    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-cd7c86/rate-test-model-cd7c86-kserve-6c9d768687-xgd2j" podUID="80888cd0-0963-4670-99d6-f33332af02f6" containerName="main" containerID="cri-o://7d84dcfe250b0fa859025e984bc0e5014a0d8c7bd0f876481475182d6e86abec" gracePeriod=45
Jul 06 21:05:34.176287 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:05:34.176220    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-cd7c86/rate-test-model-cd7c86-kserve-6c9d768687-xgd2j"
Jul 06 21:05:34.310372 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:05:34.310333    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80888cd0-0963-4670-99d6-f33332af02f6-home\") pod \"80888cd0-0963-4670-99d6-f33332af02f6\" (UID: \"80888cd0-0963-4670-99d6-f33332af02f6\") "
Jul 06 21:05:34.310372 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:05:34.310375    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80888cd0-0963-4670-99d6-f33332af02f6-tls-certs\") pod \"80888cd0-0963-4670-99d6-f33332af02f6\" (UID: \"80888cd0-0963-4670-99d6-f33332af02f6\") "
Jul 06 21:05:34.310637 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:05:34.310425    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80888cd0-0963-4670-99d6-f33332af02f6-tmp-dir\") pod \"80888cd0-0963-4670-99d6-f33332af02f6\" (UID: \"80888cd0-0963-4670-99d6-f33332af02f6\") "
Jul 06 21:05:34.310637 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:05:34.310457    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80888cd0-0963-4670-99d6-f33332af02f6-model-cache\") pod \"80888cd0-0963-4670-99d6-f33332af02f6\" (UID: \"80888cd0-0963-4670-99d6-f33332af02f6\") "
Jul 06 21:05:34.310637 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:05:34.310529    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80888cd0-0963-4670-99d6-f33332af02f6-dshm\") pod \"80888cd0-0963-4670-99d6-f33332af02f6\" (UID: \"80888cd0-0963-4670-99d6-f33332af02f6\") "
Jul 06 21:05:34.310637 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:05:34.310552    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-schwp\" (UniqueName: \"kubernetes.io/projected/80888cd0-0963-4670-99d6-f33332af02f6-kube-api-access-schwp\") pod \"80888cd0-0963-4670-99d6-f33332af02f6\" (UID: \"80888cd0-0963-4670-99d6-f33332af02f6\") "
Jul 06 21:05:34.310637 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:05:34.310620    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80888cd0-0963-4670-99d6-f33332af02f6-home" (OuterVolumeSpecName: "home") pod "80888cd0-0963-4670-99d6-f33332af02f6" (UID: "80888cd0-0963-4670-99d6-f33332af02f6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:05:34.310934 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:05:34.310767    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80888cd0-0963-4670-99d6-f33332af02f6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "80888cd0-0963-4670-99d6-f33332af02f6" (UID: "80888cd0-0963-4670-99d6-f33332af02f6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:05:34.310934 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:05:34.310789    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80888cd0-0963-4670-99d6-f33332af02f6-model-cache" (OuterVolumeSpecName: "model-cache") pod "80888cd0-0963-4670-99d6-f33332af02f6" (UID: "80888cd0-0963-4670-99d6-f33332af02f6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:05:34.310934 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:05:34.310870    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80888cd0-0963-4670-99d6-f33332af02f6-tmp-dir\") on node \"ip-10-0-141-123.ec2.internal\" DevicePath \"\""
Jul 06 21:05:34.310934 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:05:34.310890    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80888cd0-0963-4670-99d6-f33332af02f6-model-cache\") on node \"ip-10-0-141-123.ec2.internal\" DevicePath \"\""
Jul 06 21:05:34.310934 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:05:34.310902    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80888cd0-0963-4670-99d6-f33332af02f6-home\") on node \"ip-10-0-141-123.ec2.internal\" DevicePath \"\""
Jul 06 21:05:34.312638 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:05:34.312605    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80888cd0-0963-4670-99d6-f33332af02f6-kube-api-access-schwp" (OuterVolumeSpecName: "kube-api-access-schwp") pod "80888cd0-0963-4670-99d6-f33332af02f6" (UID: "80888cd0-0963-4670-99d6-f33332af02f6"). InnerVolumeSpecName "kube-api-access-schwp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 21:05:34.312638 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:05:34.312634    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80888cd0-0963-4670-99d6-f33332af02f6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "80888cd0-0963-4670-99d6-f33332af02f6" (UID: "80888cd0-0963-4670-99d6-f33332af02f6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 21:05:34.312825 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:05:34.312611    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80888cd0-0963-4670-99d6-f33332af02f6-dshm" (OuterVolumeSpecName: "dshm") pod "80888cd0-0963-4670-99d6-f33332af02f6" (UID: "80888cd0-0963-4670-99d6-f33332af02f6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:05:34.412264 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:05:34.412228    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80888cd0-0963-4670-99d6-f33332af02f6-dshm\") on node \"ip-10-0-141-123.ec2.internal\" DevicePath \"\""
Jul 06 21:05:34.412264 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:05:34.412258    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-schwp\" (UniqueName: \"kubernetes.io/projected/80888cd0-0963-4670-99d6-f33332af02f6-kube-api-access-schwp\") on node \"ip-10-0-141-123.ec2.internal\" DevicePath \"\""
Jul 06 21:05:34.412264 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:05:34.412270    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80888cd0-0963-4670-99d6-f33332af02f6-tls-certs\") on node \"ip-10-0-141-123.ec2.internal\" DevicePath \"\""
Jul 06 21:05:34.422172 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:05:34.422136    2563 generic.go:356] "Generic (PLEG): container finished" podID="80888cd0-0963-4670-99d6-f33332af02f6" containerID="7d84dcfe250b0fa859025e984bc0e5014a0d8c7bd0f876481475182d6e86abec" exitCode=0
Jul 06 21:05:34.422346 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:05:34.422183    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-cd7c86/rate-test-model-cd7c86-kserve-6c9d768687-xgd2j"
Jul 06 21:05:34.422346 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:05:34.422221    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-cd7c86/rate-test-model-cd7c86-kserve-6c9d768687-xgd2j" event={"ID":"80888cd0-0963-4670-99d6-f33332af02f6","Type":"ContainerDied","Data":"7d84dcfe250b0fa859025e984bc0e5014a0d8c7bd0f876481475182d6e86abec"}
Jul 06 21:05:34.422346 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:05:34.422268    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-cd7c86/rate-test-model-cd7c86-kserve-6c9d768687-xgd2j" event={"ID":"80888cd0-0963-4670-99d6-f33332af02f6","Type":"ContainerDied","Data":"f573cbee67dcbdda744418d84e650792db4ac40911542f5a8a8da75dd2ec1c55"}
Jul 06 21:05:34.422346 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:05:34.422293    2563 scope.go:117] "RemoveContainer" containerID="7d84dcfe250b0fa859025e984bc0e5014a0d8c7bd0f876481475182d6e86abec"
Jul 06 21:05:34.431166 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:05:34.431147    2563 scope.go:117] "RemoveContainer" containerID="7d84dcfe250b0fa859025e984bc0e5014a0d8c7bd0f876481475182d6e86abec"
Jul 06 21:05:34.431450 ip-10-0-141-123 kubenswrapper[2563]: E0706 21:05:34.431431    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d84dcfe250b0fa859025e984bc0e5014a0d8c7bd0f876481475182d6e86abec\": container with ID starting with 7d84dcfe250b0fa859025e984bc0e5014a0d8c7bd0f876481475182d6e86abec not found: ID does not exist" containerID="7d84dcfe250b0fa859025e984bc0e5014a0d8c7bd0f876481475182d6e86abec"
Jul 06 21:05:34.431496 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:05:34.431460    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d84dcfe250b0fa859025e984bc0e5014a0d8c7bd0f876481475182d6e86abec"} err="failed to get container status \"7d84dcfe250b0fa859025e984bc0e5014a0d8c7bd0f876481475182d6e86abec\": rpc error: code = NotFound desc = could not find container \"7d84dcfe250b0fa859025e984bc0e5014a0d8c7bd0f876481475182d6e86abec\": container with ID starting with 7d84dcfe250b0fa859025e984bc0e5014a0d8c7bd0f876481475182d6e86abec not found: ID does not exist"
Jul 06 21:05:34.467651 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:05:34.467620    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-cd7c86/rate-test-model-cd7c86-kserve-6c9d768687-xgd2j"]
Jul 06 21:05:34.492178 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:05:34.492149    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-cd7c86/rate-test-model-cd7c86-kserve-6c9d768687-xgd2j"]
Jul 06 21:05:34.834638 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:05:34.834600    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="80888cd0-0963-4670-99d6-f33332af02f6" path="/var/lib/kubelet/pods/80888cd0-0963-4670-99d6-f33332af02f6/volumes"
Jul 06 21:05:34.835017 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:05:34.835003    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e2d12373-de22-42a1-8871-2dd766c6b83e" path="/var/lib/kubelet/pods/e2d12373-de22-42a1-8871-2dd766c6b83e/volumes"
Jul 06 21:05:35.675186 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:05:35.675154    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 21:05:36.076473 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:05:36.076438    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 21:05:36.434783 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:05:36.434686    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 21:05:36.557199 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:05:36.557164    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 21:05:37.025956 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:05:37.025928    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 21:05:37.133268 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:05:37.133234    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 21:05:37.225676 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:05:37.225645    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 21:05:37.422896 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:05:37.422867    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 21:05:43.678465 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:05:43.678429    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 21:05:43.779869 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:05:43.779836    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 21:05:44.023379 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:05:44.023292    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 21:05:44.222957 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:05:44.222925    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 21:05:44.305211 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:05:44.305175    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-clwd5"]
Jul 06 21:05:44.624061 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:05:44.623983    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-74564584cc-w8nz7_8903f259-39c4-4473-954c-3075c773278b/authorino/0.log"
Jul 06 21:05:50.679316 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:05:50.679232    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-656f947554-6f99f_f29b869f-09ec-4619-8eb3-ab4fb32c7afb/copy-manifests/0.log"
Jul 06 21:05:50.691767 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:05:50.691713    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-656f947554-6f99f_f29b869f-09ec-4619-8eb3-ab4fb32c7afb/manager/0.log"
Jul 06 21:05:52.289142 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:05:52.289111    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-74564584cc-w8nz7_8903f259-39c4-4473-954c-3075c773278b/authorino/0.log"
Jul 06 21:05:52.428118 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:05:52.428093    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-lf8jx_70d76a1d-0156-4342-9631-c5ce326197bb/manager/0.log"
Jul 06 21:05:53.067723 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:05:53.067682    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-clwd5_f0f20c5b-abd5-41f7-bca1-24dc1b5bc457/limitador/0.log"
Jul 06 21:05:53.984532 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:05:53.984505    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-65d4f4589-nnmqp_5a6b0ea0-6a10-4a01-97c8-11d180c63547/kube-auth-proxy/0.log"
Jul 06 21:05:55.484964 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:05:55.484933    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5fc6fb456f-78mx7_a019d9e3-7fa0-4ed9-b96d-eda20e20f798/main/0.log"
Jul 06 21:05:59.507051 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:05:59.507010    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-9hcgk/must-gather-5z2ff"]
Jul 06 21:05:59.507449 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:05:59.507365    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80888cd0-0963-4670-99d6-f33332af02f6" containerName="main"
Jul 06 21:05:59.507449 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:05:59.507378    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="80888cd0-0963-4670-99d6-f33332af02f6" containerName="main"
Jul 06 21:05:59.507449 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:05:59.507400    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2d12373-de22-42a1-8871-2dd766c6b83e" containerName="main"
Jul 06 21:05:59.507449 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:05:59.507405    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2d12373-de22-42a1-8871-2dd766c6b83e" containerName="main"
Jul 06 21:05:59.507587 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:05:59.507465    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="80888cd0-0963-4670-99d6-f33332af02f6" containerName="main"
Jul 06 21:05:59.507587 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:05:59.507476    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2d12373-de22-42a1-8871-2dd766c6b83e" containerName="main"
Jul 06 21:05:59.512038 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:05:59.512015    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9hcgk/must-gather-5z2ff"
Jul 06 21:05:59.520490 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:05:59.520465    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9hcgk\"/\"kube-root-ca.crt\""
Jul 06 21:05:59.520968 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:05:59.520949    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-9hcgk\"/\"default-dockercfg-mnbcs\""
Jul 06 21:05:59.521078 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:05:59.520970    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9hcgk\"/\"openshift-service-ca.crt\""
Jul 06 21:05:59.541085 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:05:59.541036    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 21:05:59.548133 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:05:59.548098    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/872867c5-d036-4741-8e1d-7516ec91c489-must-gather-output\") pod \"must-gather-5z2ff\" (UID: \"872867c5-d036-4741-8e1d-7516ec91c489\") " pod="openshift-must-gather-9hcgk/must-gather-5z2ff"
Jul 06 21:05:59.548349 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:05:59.548145    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nkhd\" (UniqueName: \"kubernetes.io/projected/872867c5-d036-4741-8e1d-7516ec91c489-kube-api-access-6nkhd\") pod \"must-gather-5z2ff\" (UID: \"872867c5-d036-4741-8e1d-7516ec91c489\") " pod="openshift-must-gather-9hcgk/must-gather-5z2ff"
Jul 06 21:05:59.554475 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:05:59.554444    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9hcgk/must-gather-5z2ff"]
Jul 06 21:05:59.649645 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:05:59.649603    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/872867c5-d036-4741-8e1d-7516ec91c489-must-gather-output\") pod \"must-gather-5z2ff\" (UID: \"872867c5-d036-4741-8e1d-7516ec91c489\") " pod="openshift-must-gather-9hcgk/must-gather-5z2ff"
Jul 06 21:05:59.649645 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:05:59.649651    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nkhd\" (UniqueName: \"kubernetes.io/projected/872867c5-d036-4741-8e1d-7516ec91c489-kube-api-access-6nkhd\") pod \"must-gather-5z2ff\" (UID: \"872867c5-d036-4741-8e1d-7516ec91c489\") " pod="openshift-must-gather-9hcgk/must-gather-5z2ff"
Jul 06 21:05:59.649985 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:05:59.649967    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/872867c5-d036-4741-8e1d-7516ec91c489-must-gather-output\") pod \"must-gather-5z2ff\" (UID: \"872867c5-d036-4741-8e1d-7516ec91c489\") " pod="openshift-must-gather-9hcgk/must-gather-5z2ff"
Jul 06 21:05:59.662208 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:05:59.662171    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nkhd\" (UniqueName: \"kubernetes.io/projected/872867c5-d036-4741-8e1d-7516ec91c489-kube-api-access-6nkhd\") pod \"must-gather-5z2ff\" (UID: \"872867c5-d036-4741-8e1d-7516ec91c489\") " pod="openshift-must-gather-9hcgk/must-gather-5z2ff"
Jul 06 21:05:59.735409 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:05:59.735376    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log" limit=1024
Jul 06 21:05:59.821647 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:05:59.821618    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9hcgk/must-gather-5z2ff"
Jul 06 21:05:59.972681 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:05:59.972600    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9hcgk/must-gather-5z2ff"]
Jul 06 21:05:59.975170 ip-10-0-141-123 kubenswrapper[2563]: W0706 21:05:59.975136    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod872867c5_d036_4741_8e1d_7516ec91c489.slice/crio-88e09023dfd7503331d58a0605af458b8d7623b89892c9f23320ea92ca857938 WatchSource:0}: Error finding container 88e09023dfd7503331d58a0605af458b8d7623b89892c9f23320ea92ca857938: Status 404 returned error can't find the container with id 88e09023dfd7503331d58a0605af458b8d7623b89892c9f23320ea92ca857938
Jul 06 21:05:59.977216 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:05:59.977192    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 06 21:06:00.550100 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:06:00.550058    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9hcgk/must-gather-5z2ff" event={"ID":"872867c5-d036-4741-8e1d-7516ec91c489","Type":"ContainerStarted","Data":"88e09023dfd7503331d58a0605af458b8d7623b89892c9f23320ea92ca857938"}
Jul 06 21:06:01.563862 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:06:01.563182    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9hcgk/must-gather-5z2ff" event={"ID":"872867c5-d036-4741-8e1d-7516ec91c489","Type":"ContainerStarted","Data":"e4d7e7aab87b5b19670fc34abbc009c3453108c74d40127276e9313789226bb8"}
Jul 06 21:06:01.563862 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:06:01.563240    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9hcgk/must-gather-5z2ff" event={"ID":"872867c5-d036-4741-8e1d-7516ec91c489","Type":"ContainerStarted","Data":"39c708edb5f7025d34854748e0aeec7551b8babc438c07681ed9a724ec290abb"}
Jul 06 21:06:01.592577 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:06:01.592509    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-9hcgk/must-gather-5z2ff" podStartSLOduration=1.7657769079999999 podStartE2EDuration="2.592485572s" totalImagesPullingTime="826.708664ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:05:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:06:01.591250272 +0000 UTC m=+2525.307360959" watchObservedRunningTime="2026-07-06 21:06:01.592485572 +0000 UTC m=+2525.308596233"
Jul 06 21:06:02.751399 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:06:02.751370    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-nzf7w_e42d3c32-18ba-48dc-afb5-90ef7f0e2752/global-pull-secret-syncer/0.log"
Jul 06 21:06:02.940164 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:06:02.940133    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-2cqxl_408f722a-5e7e-4bad-a86e-d461b5fed747/konnectivity-agent/0.log"
Jul 06 21:06:03.128870 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:06:03.128835    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-123.ec2.internal_bc8b346ff26bd2da637db45021bfde3d/haproxy/0.log"
Jul 06 21:06:08.091756 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:06:08.091012    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-74564584cc-w8nz7_8903f259-39c4-4473-954c-3075c773278b/authorino/0.log"
Jul 06 21:06:08.456261 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:06:08.456229    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-lf8jx_70d76a1d-0156-4342-9631-c5ce326197bb/manager/0.log"
Jul 06 21:06:08.880882 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:06:08.880849    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-clwd5_f0f20c5b-abd5-41f7-bca1-24dc1b5bc457/limitador/0.log"
Jul 06 21:06:10.612830 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:06:10.612800    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f571db4e-af94-4e27-9e81-3575f4620e69/alertmanager/0.log"
Jul 06 21:06:10.652695 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:06:10.652649    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f571db4e-af94-4e27-9e81-3575f4620e69/config-reloader/0.log"
Jul 06 21:06:10.695197 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:06:10.695174    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f571db4e-af94-4e27-9e81-3575f4620e69/kube-rbac-proxy-web/0.log"
Jul 06 21:06:10.742958 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:06:10.742926    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f571db4e-af94-4e27-9e81-3575f4620e69/kube-rbac-proxy/0.log"
Jul 06 21:06:10.813963 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:06:10.813925    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f571db4e-af94-4e27-9e81-3575f4620e69/kube-rbac-proxy-metric/0.log"
Jul 06 21:06:10.859432 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:06:10.859381    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f571db4e-af94-4e27-9e81-3575f4620e69/prom-label-proxy/0.log"
Jul 06 21:06:10.911262 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:06:10.911172    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f571db4e-af94-4e27-9e81-3575f4620e69/init-config-reloader/0.log"
Jul 06 21:06:11.042309 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:06:11.042275    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-88wl5_d10133bf-b0d1-4ce9-9906-59fbafad0cc5/kube-state-metrics/0.log"
Jul 06 21:06:11.108889 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:06:11.108853    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-88wl5_d10133bf-b0d1-4ce9-9906-59fbafad0cc5/kube-rbac-proxy-main/0.log"
Jul 06 21:06:11.173966 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:06:11.173848    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-88wl5_d10133bf-b0d1-4ce9-9906-59fbafad0cc5/kube-rbac-proxy-self/0.log"
Jul 06 21:06:11.248402 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:06:11.248373    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-5b84f65c46-pb5v8_6827cc99-95c6-4a25-8e25-e73e7e3e2d7c/metrics-server/0.log"
Jul 06 21:06:11.316914 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:06:11.316871    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-d5dcff7cf-m2dqb_3be44d9c-34b9-4f83-b328-de358e619205/monitoring-plugin/0.log"
Jul 06 21:06:11.374546 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:06:11.374511    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kc45l_2640cc2d-7e6f-4eb0-a176-eb62a152789a/node-exporter/0.log"
Jul 06 21:06:11.407376 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:06:11.407334    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kc45l_2640cc2d-7e6f-4eb0-a176-eb62a152789a/kube-rbac-proxy/0.log"
Jul 06 21:06:11.452766 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:06:11.452651    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kc45l_2640cc2d-7e6f-4eb0-a176-eb62a152789a/init-textfile/0.log"
Jul 06 21:06:12.097029 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:06:12.096995    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a62a9219-ae33-42ff-800a-4fd7c8b9c908/prometheus/0.log"
Jul 06 21:06:12.130754 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:06:12.130710    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a62a9219-ae33-42ff-800a-4fd7c8b9c908/config-reloader/0.log"
Jul 06 21:06:12.169268 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:06:12.169238    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a62a9219-ae33-42ff-800a-4fd7c8b9c908/thanos-sidecar/0.log"
Jul 06 21:06:12.202382 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:06:12.202356    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a62a9219-ae33-42ff-800a-4fd7c8b9c908/kube-rbac-proxy-web/0.log"
Jul 06 21:06:12.232326 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:06:12.232294    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a62a9219-ae33-42ff-800a-4fd7c8b9c908/kube-rbac-proxy/0.log"
Jul 06 21:06:12.262095 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:06:12.262067    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a62a9219-ae33-42ff-800a-4fd7c8b9c908/kube-rbac-proxy-thanos/0.log"
Jul 06 21:06:12.292760 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:06:12.292714    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a62a9219-ae33-42ff-800a-4fd7c8b9c908/init-config-reloader/0.log"
Jul 06 21:06:12.340786 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:06:12.340755    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-567f7b8b7c-qphwl_b33528c3-08cc-4595-aa46-8a72187a353d/prometheus-operator/0.log"
Jul 06 21:06:12.380185 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:06:12.380098    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-567f7b8b7c-qphwl_b33528c3-08cc-4595-aa46-8a72187a353d/kube-rbac-proxy/0.log"
Jul 06 21:06:12.998497 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:06:12.998458    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-9hcgk/perf-node-gather-daemonset-cg5r4"]
Jul 06 21:06:13.004646 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:06:13.004616    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9hcgk/perf-node-gather-daemonset-cg5r4"
Jul 06 21:06:13.018619 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:06:13.018532    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9hcgk/perf-node-gather-daemonset-cg5r4"]
Jul 06 21:06:13.042855 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:06:13.042811    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f9f26c45-69c9-4089-a7ed-94498685812b-sys\") pod \"perf-node-gather-daemonset-cg5r4\" (UID: \"f9f26c45-69c9-4089-a7ed-94498685812b\") " pod="openshift-must-gather-9hcgk/perf-node-gather-daemonset-cg5r4"
Jul 06 21:06:13.043153 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:06:13.043129    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f9f26c45-69c9-4089-a7ed-94498685812b-proc\") pod \"perf-node-gather-daemonset-cg5r4\" (UID: \"f9f26c45-69c9-4089-a7ed-94498685812b\") " pod="openshift-must-gather-9hcgk/perf-node-gather-daemonset-cg5r4"
Jul 06 21:06:13.043408 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:06:13.043388    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h85jc\" (UniqueName: \"kubernetes.io/projected/f9f26c45-69c9-4089-a7ed-94498685812b-kube-api-access-h85jc\") pod \"perf-node-gather-daemonset-cg5r4\" (UID: \"f9f26c45-69c9-4089-a7ed-94498685812b\") " pod="openshift-must-gather-9hcgk/perf-node-gather-daemonset-cg5r4"
Jul 06 21:06:13.043571 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:06:13.043555    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f9f26c45-69c9-4089-a7ed-94498685812b-podres\") pod \"perf-node-gather-daemonset-cg5r4\" (UID: \"f9f26c45-69c9-4089-a7ed-94498685812b\") " pod="openshift-must-gather-9hcgk/perf-node-gather-daemonset-cg5r4"
Jul 06 21:06:13.043767 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:06:13.043749    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f9f26c45-69c9-4089-a7ed-94498685812b-lib-modules\") pod \"perf-node-gather-daemonset-cg5r4\" (UID: \"f9f26c45-69c9-4089-a7ed-94498685812b\") " pod="openshift-must-gather-9hcgk/perf-node-gather-daemonset-cg5r4"
Jul 06 21:06:13.144314 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:06:13.144280    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f9f26c45-69c9-4089-a7ed-94498685812b-lib-modules\") pod \"perf-node-gather-daemonset-cg5r4\" (UID: \"f9f26c45-69c9-4089-a7ed-94498685812b\") " pod="openshift-must-gather-9hcgk/perf-node-gather-daemonset-cg5r4"
Jul 06 21:06:13.144817 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:06:13.144333    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f9f26c45-69c9-4089-a7ed-94498685812b-sys\") pod \"perf-node-gather-daemonset-cg5r4\" (UID: \"f9f26c45-69c9-4089-a7ed-94498685812b\") " pod="openshift-must-gather-9hcgk/perf-node-gather-daemonset-cg5r4"
Jul 06 21:06:13.144817 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:06:13.144350    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f9f26c45-69c9-4089-a7ed-94498685812b-proc\") pod \"perf-node-gather-daemonset-cg5r4\" (UID: \"f9f26c45-69c9-4089-a7ed-94498685812b\") " pod="openshift-must-gather-9hcgk/perf-node-gather-daemonset-cg5r4"
Jul 06 21:06:13.144817 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:06:13.144397    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h85jc\" (UniqueName: \"kubernetes.io/projected/f9f26c45-69c9-4089-a7ed-94498685812b-kube-api-access-h85jc\") pod \"perf-node-gather-daemonset-cg5r4\" (UID: \"f9f26c45-69c9-4089-a7ed-94498685812b\") " pod="openshift-must-gather-9hcgk/perf-node-gather-daemonset-cg5r4"
Jul 06 21:06:13.144817 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:06:13.144421    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f9f26c45-69c9-4089-a7ed-94498685812b-podres\") pod \"perf-node-gather-daemonset-cg5r4\" (UID: \"f9f26c45-69c9-4089-a7ed-94498685812b\") " pod="openshift-must-gather-9hcgk/perf-node-gather-daemonset-cg5r4"
Jul 06 21:06:13.144817 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:06:13.144417    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f9f26c45-69c9-4089-a7ed-94498685812b-sys\") pod \"perf-node-gather-daemonset-cg5r4\" (UID: \"f9f26c45-69c9-4089-a7ed-94498685812b\") " pod="openshift-must-gather-9hcgk/perf-node-gather-daemonset-cg5r4"
Jul 06 21:06:13.144817 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:06:13.144460    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f9f26c45-69c9-4089-a7ed-94498685812b-lib-modules\") pod \"perf-node-gather-daemonset-cg5r4\" (UID: \"f9f26c45-69c9-4089-a7ed-94498685812b\") " pod="openshift-must-gather-9hcgk/perf-node-gather-daemonset-cg5r4"
Jul 06 21:06:13.144817 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:06:13.144542    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f9f26c45-69c9-4089-a7ed-94498685812b-podres\") pod \"perf-node-gather-daemonset-cg5r4\" (UID: \"f9f26c45-69c9-4089-a7ed-94498685812b\") " pod="openshift-must-gather-9hcgk/perf-node-gather-daemonset-cg5r4"
Jul 06 21:06:13.144817 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:06:13.144627    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f9f26c45-69c9-4089-a7ed-94498685812b-proc\") pod \"perf-node-gather-daemonset-cg5r4\" (UID: \"f9f26c45-69c9-4089-a7ed-94498685812b\") " pod="openshift-must-gather-9hcgk/perf-node-gather-daemonset-cg5r4"
Jul 06 21:06:13.156784 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:06:13.156748    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h85jc\" (UniqueName: \"kubernetes.io/projected/f9f26c45-69c9-4089-a7ed-94498685812b-kube-api-access-h85jc\") pod \"perf-node-gather-daemonset-cg5r4\" (UID: \"f9f26c45-69c9-4089-a7ed-94498685812b\") " pod="openshift-must-gather-9hcgk/perf-node-gather-daemonset-cg5r4"
Jul 06 21:06:13.320667 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:06:13.320598    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9hcgk/perf-node-gather-daemonset-cg5r4"
Jul 06 21:06:13.526323 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:06:13.526288    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9hcgk/perf-node-gather-daemonset-cg5r4"]
Jul 06 21:06:13.529465 ip-10-0-141-123 kubenswrapper[2563]: W0706 21:06:13.529427    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podf9f26c45_69c9_4089_a7ed_94498685812b.slice/crio-f2b6463fe22bda8c07d4f865699da2b9b83af85f73aa9fa70317524a97950723 WatchSource:0}: Error finding container f2b6463fe22bda8c07d4f865699da2b9b83af85f73aa9fa70317524a97950723: Status 404 returned error can't find the container with id f2b6463fe22bda8c07d4f865699da2b9b83af85f73aa9fa70317524a97950723
Jul 06 21:06:13.646141 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:06:13.646105    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9hcgk/perf-node-gather-daemonset-cg5r4" event={"ID":"f9f26c45-69c9-4089-a7ed-94498685812b","Type":"ContainerStarted","Data":"f2b6463fe22bda8c07d4f865699da2b9b83af85f73aa9fa70317524a97950723"}
Jul 06 21:06:14.646421 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:06:14.646393    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-2m7gh_bb2a41aa-a9fc-4ec5-9b78-5e22c1fd301b/console-operator/1.log"
Jul 06 21:06:14.652663 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:06:14.652633    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9hcgk/perf-node-gather-daemonset-cg5r4" event={"ID":"f9f26c45-69c9-4089-a7ed-94498685812b","Type":"ContainerStarted","Data":"b22c978d56c7053719cabf1b3b04896da80924674332746c3615c3f8895748a0"}
Jul 06 21:06:14.652866 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:06:14.652743    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-9hcgk/perf-node-gather-daemonset-cg5r4"
Jul 06 21:06:14.652866 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:06:14.652790    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-2m7gh_bb2a41aa-a9fc-4ec5-9b78-5e22c1fd301b/console-operator/2.log"
Jul 06 21:06:14.693649 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:06:14.693568    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-9hcgk/perf-node-gather-daemonset-cg5r4" podStartSLOduration=2.6935448749999997 podStartE2EDuration="2.693544875s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 21:06:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:06:14.691151811 +0000 UTC m=+2538.407262473" watchObservedRunningTime="2026-07-06 21:06:14.693544875 +0000 UTC m=+2538.409655591"
Jul 06 21:06:15.297519 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:06:15.297482    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-65f6bc98bb-d9fxk_66f2e86a-a524-469e-98d2-49ea841709f4/console/0.log"
Jul 06 21:06:16.005060 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:06:16.005032    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5d456fdbcd-tcw8f_c8b77a06-e378-4539-9e89-d0e6209b2da5/volume-data-source-validator/0.log"
Jul 06 21:06:17.546873 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:06:17.546835    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-gqvcm_f2c17640-14e9-4b68-93a4-9183cbc2ccd0/dns/0.log"
Jul 06 21:06:17.588091 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:06:17.588063    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-gqvcm_f2c17640-14e9-4b68-93a4-9183cbc2ccd0/kube-rbac-proxy/0.log"
Jul 06 21:06:17.693650 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:06:17.693623    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-n9568_1816531c-ac4a-4d49-96ac-234daa36ea91/dns-node-resolver/0.log"
Jul 06 21:06:18.491567 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:06:18.491540    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-2hwd9_ed41c82c-bbcc-4385-8f89-f6ff340c2e36/node-ca/0.log"
Jul 06 21:06:19.818928 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:06:19.818892    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-65d4f4589-nnmqp_5a6b0ea0-6a10-4a01-97c8-11d180c63547/kube-auth-proxy/0.log"
Jul 06 21:06:20.668998 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:06:20.668971    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-9hcgk/perf-node-gather-daemonset-cg5r4"
Jul 06 21:06:20.942345 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:06:20.942236    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-fn7x6_3cce0a2d-3fc6-4667-b38a-a0e800fbfe0b/serve-healthcheck-canary/0.log"
Jul 06 21:06:21.660014 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:06:21.659988    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hg45g_abbed80c-bca8-49ce-91e5-692152f3d0e5/kube-rbac-proxy/0.log"
Jul 06 21:06:21.702592 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:06:21.702556    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hg45g_abbed80c-bca8-49ce-91e5-692152f3d0e5/exporter/0.log"
Jul 06 21:06:21.738073 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:06:21.738043    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hg45g_abbed80c-bca8-49ce-91e5-692152f3d0e5/extractor/0.log"
Jul 06 21:06:24.175603 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:06:24.175569    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-65d4f4589-nnmqp_5a6b0ea0-6a10-4a01-97c8-11d180c63547/kube-auth-proxy/0.log"
Jul 06 21:06:25.442828 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:06:25.442756    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-656f947554-6f99f_f29b869f-09ec-4619-8eb3-ab4fb32c7afb/manager/0.log"
Jul 06 21:06:25.511646 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:06:25.511620    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-656f947554-6f99f_f29b869f-09ec-4619-8eb3-ab4fb32c7afb/copy-manifests/0.log"
Jul 06 21:06:25.713241 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:06:25.713154    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-65d4f4589-nnmqp_5a6b0ea0-6a10-4a01-97c8-11d180c63547/kube-auth-proxy/0.log"
Jul 06 21:06:26.997483 ip-10-0-141-123 kubenswrapper[2563]: I0706 21:06:26.997447    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-6cf865c78f-phbhx_2e486719-5dc2-48d2-ac78-293db8913c7c/manager/0.log"