Jul 12 17:01:08.829947 ip-10-0-141-94 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 12 17:01:08.829958 ip-10-0-141-94 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 12 17:01:08.829967 ip-10-0-141-94 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 12 17:01:08.830272 ip-10-0-141-94 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 12 17:01:19.049042 ip-10-0-141-94 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 12 17:01:19.049062 ip-10-0-141-94 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot b28f51f260b040de809063821c51168e --
Jul 12 17:03:10.694981 ip-10-0-141-94 systemd[1]: Starting Kubernetes Kubelet...
Jul 12 17:03:11.073752 ip-10-0-141-94 kubenswrapper[2569]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 12 17:03:11.073752 ip-10-0-141-94 kubenswrapper[2569]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 12 17:03:11.073752 ip-10-0-141-94 kubenswrapper[2569]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 12 17:03:11.073752 ip-10-0-141-94 kubenswrapper[2569]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 12 17:03:11.073752 ip-10-0-141-94 kubenswrapper[2569]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 12 17:03:11.076222 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.076151    2569 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 12 17:03:11.080932 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.080918    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 12 17:03:11.080932 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.080932    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 12 17:03:11.080998 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.080936    2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 12 17:03:11.080998 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.080939    2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 12 17:03:11.080998 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.080943    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 12 17:03:11.080998 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.080946    2569 feature_gate.go:328] unrecognized feature gate: Example2
Jul 12 17:03:11.080998 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.080949    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 12 17:03:11.080998 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.080952    2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 12 17:03:11.080998 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.080954    2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 12 17:03:11.080998 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.080957    2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 12 17:03:11.080998 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.080960    2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 12 17:03:11.080998 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.080962    2569 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 12 17:03:11.080998 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.080968    2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 12 17:03:11.080998 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.080972    2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 12 17:03:11.080998 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.080975    2569 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 12 17:03:11.080998 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.080979    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 12 17:03:11.080998 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.080982    2569 feature_gate.go:328] unrecognized feature gate: Example
Jul 12 17:03:11.080998 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.080986    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 12 17:03:11.080998 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.080989    2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 12 17:03:11.080998 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.080992    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 12 17:03:11.080998 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.080995    2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 12 17:03:11.080998 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.080998    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 12 17:03:11.081469 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.081001    2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 12 17:03:11.081469 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.081004    2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 12 17:03:11.081469 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.081006    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 12 17:03:11.081469 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.081009    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 12 17:03:11.081469 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.081012    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 12 17:03:11.081469 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.081015    2569 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 12 17:03:11.081469 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.081018    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 12 17:03:11.081469 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.081020    2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 12 17:03:11.081469 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.081023    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 12 17:03:11.081469 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.081026    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 12 17:03:11.081469 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.081028    2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 12 17:03:11.081469 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.081033    2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 12 17:03:11.081469 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.081036    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 12 17:03:11.081469 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.081038    2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 12 17:03:11.081469 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.081041    2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 12 17:03:11.081469 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.081043    2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 12 17:03:11.081469 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.081046    2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 12 17:03:11.081469 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.081049    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 12 17:03:11.081469 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.081051    2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 12 17:03:11.081974 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.081054    2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 12 17:03:11.081974 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.081057    2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 12 17:03:11.081974 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.081061    2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 12 17:03:11.081974 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.081063    2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 12 17:03:11.081974 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.081066    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 12 17:03:11.081974 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.081068    2569 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 12 17:03:11.081974 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.081071    2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 12 17:03:11.081974 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.081073    2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 12 17:03:11.081974 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.081075    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 12 17:03:11.081974 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.081078    2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 12 17:03:11.081974 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.081080    2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 12 17:03:11.081974 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.081083    2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 12 17:03:11.081974 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.081085    2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 12 17:03:11.081974 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.081088    2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 12 17:03:11.081974 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.081091    2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 12 17:03:11.081974 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.081095    2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 12 17:03:11.081974 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.081098    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 12 17:03:11.081974 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.081100    2569 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 12 17:03:11.081974 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.081103    2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 12 17:03:11.081974 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.081106    2569 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 12 17:03:11.082461 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.081108    2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 12 17:03:11.082461 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.081111    2569 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 12 17:03:11.082461 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.081114    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 12 17:03:11.082461 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.081116    2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 12 17:03:11.082461 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.081119    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 12 17:03:11.082461 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.081121    2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 12 17:03:11.082461 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.081124    2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 12 17:03:11.082461 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.081127    2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 12 17:03:11.082461 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.081129    2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 12 17:03:11.082461 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.081131    2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 12 17:03:11.082461 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.081134    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 12 17:03:11.082461 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.081137    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 12 17:03:11.082461 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.081139    2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 12 17:03:11.082461 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.081142    2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 12 17:03:11.082461 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.081145    2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 12 17:03:11.082461 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.081148    2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 12 17:03:11.082461 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.081151    2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 12 17:03:11.082461 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.081153    2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 12 17:03:11.082461 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.081156    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 12 17:03:11.082461 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.081159    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 12 17:03:11.082953 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.081161    2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 12 17:03:11.082953 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.081164    2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 12 17:03:11.082953 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.081166    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 12 17:03:11.082953 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.081169    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 12 17:03:11.082953 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.081171    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 12 17:03:11.082953 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.081175    2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 12 17:03:11.082953 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.081178    2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 12 17:03:11.082953 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.081180    2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 12 17:03:11.082953 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.081183    2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 12 17:03:11.082953 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.081185    2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 12 17:03:11.082953 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.081189    2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 12 17:03:11.082953 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.081193    2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 12 17:03:11.082953 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.081195    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 12 17:03:11.082953 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.081199    2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 12 17:03:11.082953 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.081202    2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 12 17:03:11.082953 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.081204    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 12 17:03:11.082953 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.081206    2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 12 17:03:11.082953 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.081209    2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 12 17:03:11.082953 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.081891    2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 12 17:03:11.083405 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.081897    2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 12 17:03:11.083405 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.081900    2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 12 17:03:11.083405 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.081903    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 12 17:03:11.083405 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.081906    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 12 17:03:11.083405 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.081908    2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 12 17:03:11.083405 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.081911    2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 12 17:03:11.083405 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.081913    2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 12 17:03:11.083405 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.081916    2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 12 17:03:11.083405 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.081919    2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 12 17:03:11.083405 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.081922    2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 12 17:03:11.083405 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.081924    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 12 17:03:11.083405 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.081927    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 12 17:03:11.083405 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.081931    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 12 17:03:11.083405 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.081934    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 12 17:03:11.083405 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.081937    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 12 17:03:11.083405 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.081939    2569 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 12 17:03:11.083405 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.081942    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 12 17:03:11.083405 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.081945    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 12 17:03:11.083405 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.081948    2569 feature_gate.go:328] unrecognized feature gate: Example
Jul 12 17:03:11.083405 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.081951    2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 12 17:03:11.083911 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.081953    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 12 17:03:11.083911 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.081956    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 12 17:03:11.083911 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.081958    2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 12 17:03:11.083911 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.081961    2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 12 17:03:11.083911 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.081963    2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 12 17:03:11.083911 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.081966    2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 12 17:03:11.083911 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.081970    2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 12 17:03:11.083911 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.081974    2569 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 12 17:03:11.083911 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.081977    2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 12 17:03:11.083911 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.081980    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 12 17:03:11.083911 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.081982    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 12 17:03:11.083911 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.081985    2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 12 17:03:11.083911 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.081989    2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 12 17:03:11.083911 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.081992    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 12 17:03:11.083911 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.081995    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 12 17:03:11.083911 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.081998    2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 12 17:03:11.083911 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.082001    2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 12 17:03:11.083911 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.082003    2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 12 17:03:11.084343 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.082006    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 12 17:03:11.084343 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.082009    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 12 17:03:11.084343 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.082011    2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 12 17:03:11.084343 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.082014    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 12 17:03:11.084343 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.082016    2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 12 17:03:11.084343 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.082019    2569 feature_gate.go:328] unrecognized feature gate: Example2
Jul 12 17:03:11.084343 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.082022    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 12 17:03:11.084343 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.082025    2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 12 17:03:11.084343 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.082028    2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 12 17:03:11.084343 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.082031    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 12 17:03:11.084343 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.082034    2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 12 17:03:11.084343 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.082036    2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 12 17:03:11.084343 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.082039    2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 12 17:03:11.084343 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.082041    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 12 17:03:11.084343 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.082044    2569 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 12 17:03:11.084343 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.082046    2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 12 17:03:11.084343 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.082049    2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 12 17:03:11.084343 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.082052    2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 12 17:03:11.084343 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.082054    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 12 17:03:11.084343 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.082057    2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 12 17:03:11.084855 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.082059    2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 12 17:03:11.084855 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.082061    2569 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 12 17:03:11.084855 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.082064    2569 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 12 17:03:11.084855 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.082067    2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 12 17:03:11.084855 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.082069    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 12 17:03:11.084855 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.082072    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 12 17:03:11.084855 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.082074    2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 12 17:03:11.084855 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.082077    2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 12 17:03:11.084855 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.082079    2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 12 17:03:11.084855 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.082082    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 12 17:03:11.084855 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.082084    2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 12 17:03:11.084855 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.082087    2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 12 17:03:11.084855 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.082089    2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 12 17:03:11.084855 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.082092    2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 12 17:03:11.084855 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.082094    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 12 17:03:11.084855 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.082096    2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 12 17:03:11.084855 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.082099    2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 12 17:03:11.084855 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.082102    2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 12 17:03:11.084855 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.082105    2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 12 17:03:11.084855 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.082109    2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 12 17:03:11.085362 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.082113    2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 12 17:03:11.085362 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.082115    2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 12 17:03:11.085362 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.082118    2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 12 17:03:11.085362 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.082121    2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 12 17:03:11.085362 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.082123    2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 12 17:03:11.085362 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.082126    2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 12 17:03:11.085362 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.082129    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 12 17:03:11.085362 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.082131    2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 12 17:03:11.085362 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.082134    2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 12 17:03:11.085362 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.082136    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 12 17:03:11.085362 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.082139    2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 12 17:03:11.085362 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.082141    2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 12 17:03:11.085362 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.082144    2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 12 17:03:11.085362 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.082146    2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 12 17:03:11.085362 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.082149    2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 12 17:03:11.085362 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.082152    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 12 17:03:11.085362 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.082154    2569 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 12 17:03:11.085362 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.082157    2569 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 12 17:03:11.085362 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.082159    2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 12 17:03:11.085362 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.082162    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 12 17:03:11.085872 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.083628    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 12 17:03:11.085872 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.083636    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 12 17:03:11.085872 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.083639    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 12 17:03:11.085872 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.083642    2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 12 17:03:11.085872 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.083646    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 12 17:03:11.085872 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.083649    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 12 17:03:11.085872 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.083651    2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 12 17:03:11.085872 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.083654    2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 12 17:03:11.085872 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.083657    2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 12 17:03:11.085872 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.083661    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 12 17:03:11.085872 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.083664    2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 12 17:03:11.085872 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.083666    2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 12 17:03:11.085872 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.083669    2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 12 17:03:11.085872 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.083672    2569 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 12 17:03:11.085872 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.083676    2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 12 17:03:11.085872 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.083680    2569 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 12 17:03:11.085872 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.083682    2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 12 17:03:11.085872 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.083685    2569 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 12 17:03:11.085872 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.083688    2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 12 17:03:11.086327 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.083691    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 12 17:03:11.086327 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.083694    2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 12 17:03:11.086327 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.083697    2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 12 17:03:11.086327 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.083700    2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 12 17:03:11.086327 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.083717    2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 12 17:03:11.086327 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.083721    2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 12 17:03:11.086327 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.083726    2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 12 17:03:11.086327 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.083731    2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 12 17:03:11.086327 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.083735    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 12 17:03:11.086327 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.083739    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 12 17:03:11.086327 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.083742    2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 12 17:03:11.086327 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.083745    2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 12 17:03:11.086327 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.083747    2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 12 17:03:11.086327 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.083750    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 12 17:03:11.086327 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.083754    2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 12 17:03:11.086327 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.083756    2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 12 17:03:11.086327 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.083759    2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 12 17:03:11.086327 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.083762    2569 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 12 17:03:11.086327 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.083764    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 12 17:03:11.086327 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.083767    2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 12 17:03:11.086861 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.083769    2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 12 17:03:11.086861 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.083772    2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 12 17:03:11.086861 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.083775    2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 12 17:03:11.086861 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.083778    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 12 17:03:11.086861 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.083781    2569 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 12 17:03:11.086861 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.083784    2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 12 17:03:11.086861 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.083787    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 12 17:03:11.086861 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.083789    2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 12 17:03:11.086861 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.083792    2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 12 17:03:11.086861 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.083794    2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 12 17:03:11.086861 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.083798    2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 12 17:03:11.086861 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.083801    2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 12 17:03:11.086861 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.083804    2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 12 17:03:11.086861 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.083807    2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 12 17:03:11.086861 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.083809    2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 12 17:03:11.086861 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.083812    2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 12 17:03:11.086861 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.083815    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 12 17:03:11.086861 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.083817    2569 feature_gate.go:328] unrecognized feature gate: Example
Jul 12 17:03:11.086861 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.083820    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 12 17:03:11.086861 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.083823    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 12 17:03:11.087338 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.083825    2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 12 17:03:11.087338 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.083828    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 12 17:03:11.087338 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.083830    2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 12 17:03:11.087338 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.083833    2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 12 17:03:11.087338 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.083835    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 12 17:03:11.087338 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.083838    2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 12 17:03:11.087338 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.083840    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 12 17:03:11.087338 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.083843    2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 12 17:03:11.087338 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.083847    2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 12 17:03:11.087338 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.083851    2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 12 17:03:11.087338 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.083854    2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 12 17:03:11.087338 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.083856    2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 12 17:03:11.087338 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.083859    2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 12 17:03:11.087338 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.083862    2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 12 17:03:11.087338 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.083865    2569 feature_gate.go:328] unrecognized feature gate: Example2
Jul 12 17:03:11.087338 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.083867    2569 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 12 17:03:11.087338 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.083871    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 12 17:03:11.087338 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.083873    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 12 17:03:11.087338 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.083875    2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 12 17:03:11.087813 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.083878    2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 12 17:03:11.087813 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.083880    2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 12 17:03:11.087813 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.083883    2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 12 17:03:11.087813 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.083886    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 12 17:03:11.087813 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.083889    2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 12 17:03:11.087813 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.083891    2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 12 17:03:11.087813 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.083894    2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 12 17:03:11.087813 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.083896    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 12 17:03:11.087813 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.083898    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 12 17:03:11.087813 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.083901    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 12 17:03:11.087813 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.083903    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 12 17:03:11.087813 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.083906    2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 12 17:03:11.087813 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.083908    2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 12 17:03:11.087813 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.083911    2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 12 17:03:11.087813 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.083914    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 12 17:03:11.087813 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.083917    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 12 17:03:11.087813 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.083919    2569 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 12 17:03:11.087813 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.083922    2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 12 17:03:11.087813 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.083924    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 12 17:03:11.088310 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.083927    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 12 17:03:11.088310 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.083929    2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 12 17:03:11.088310 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.083992    2569 flags.go:64] FLAG: --address="0.0.0.0"
Jul 12 17:03:11.088310 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.084003    2569 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 12 17:03:11.088310 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.084008    2569 flags.go:64] FLAG: --anonymous-auth="true"
Jul 12 17:03:11.088310 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.084012    2569 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 12 17:03:11.088310 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.084016    2569 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 12 17:03:11.088310 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.084020    2569 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 12 17:03:11.088310 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.084024    2569 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 12 17:03:11.088310 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.084029    2569 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 12 17:03:11.088310 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.084032    2569 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 12 17:03:11.088310 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.084036    2569 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 12 17:03:11.088310 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.084040    2569 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 12 17:03:11.088310 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.084043    2569 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 12 17:03:11.088310 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.084046    2569 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 12 17:03:11.088310 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.084049    2569 flags.go:64] FLAG: --cgroup-root=""
Jul 12 17:03:11.088310 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.084052    2569 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 12 17:03:11.088310 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.084055    2569 flags.go:64] FLAG: --client-ca-file=""
Jul 12 17:03:11.088310 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.084058    2569 flags.go:64] FLAG: --cloud-provider="external"
Jul 12 17:03:11.088310 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.084061    2569 flags.go:64] FLAG: --cluster-dns="[]"
Jul 12 17:03:11.088310 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.084066    2569 flags.go:64] FLAG: --cluster-domain=""
Jul 12 17:03:11.088310 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.084068    2569 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 12 17:03:11.088310 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.084071    2569 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 12 17:03:11.088310 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.084074    2569 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 12 17:03:11.088904 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.084078    2569 flags.go:64] FLAG: --container-log-max-files="5"
Jul 12 17:03:11.088904 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.084081    2569 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 12 17:03:11.088904 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.084085    2569 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 12 17:03:11.088904 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.084088    2569 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 12 17:03:11.088904 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.084091    2569 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 12 17:03:11.088904 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.084094    2569 flags.go:64] FLAG: --contention-profiling="false"
Jul 12 17:03:11.088904 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.084097    2569 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 12 17:03:11.088904 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.084100    2569 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 12 17:03:11.088904 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.084103    2569 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 12 17:03:11.088904 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.084106    2569 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 12 17:03:11.088904 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.084110    2569 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 12 17:03:11.088904 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.084113    2569 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 12 17:03:11.088904 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.084116    2569 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 12 17:03:11.088904 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.084118    2569 flags.go:64] FLAG: --enable-load-reader="false"
Jul 12 17:03:11.088904 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.084121    2569 flags.go:64] FLAG: --enable-server="true"
Jul 12 17:03:11.088904 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.084126    2569 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 12 17:03:11.088904 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.084131    2569 flags.go:64] FLAG: --event-burst="100"
Jul 12 17:03:11.088904 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.084134    2569 flags.go:64] FLAG: --event-qps="50"
Jul 12 17:03:11.088904 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.084137    2569 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 12 17:03:11.088904 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.084148    2569 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 12 17:03:11.088904 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.084152    2569 flags.go:64] FLAG: --eviction-hard=""
Jul 12 17:03:11.088904 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.084156    2569 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 12 17:03:11.088904 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.084159    2569 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 12 17:03:11.088904 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.084162    2569 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 12 17:03:11.088904 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.084165    2569 flags.go:64] FLAG: --eviction-soft=""
Jul 12 17:03:11.089511 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.084168    2569 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 12 17:03:11.089511 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.084171    2569 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 12 17:03:11.089511 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.084174    2569 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 12 17:03:11.089511 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.084177    2569 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 12 17:03:11.089511 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.084180    2569 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 12 17:03:11.089511 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.084182    2569 flags.go:64] FLAG: --fail-swap-on="true"
Jul 12 17:03:11.089511 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.084185    2569 flags.go:64] FLAG: --feature-gates=""
Jul 12 17:03:11.089511 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.084189    2569 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 12 17:03:11.089511 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.084191    2569 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 12 17:03:11.089511 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.084194    2569 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 12 17:03:11.089511 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.084197    2569 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 12 17:03:11.089511 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.084200    2569 flags.go:64] FLAG: --healthz-port="10248"
Jul 12 17:03:11.089511 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.084203    2569 flags.go:64] FLAG: --help="false"
Jul 12 17:03:11.089511 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.084206    2569 flags.go:64] FLAG: --hostname-override="ip-10-0-141-94.ec2.internal"
Jul 12 17:03:11.089511 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.084209    2569 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 12 17:03:11.089511 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.084211    2569 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 12 17:03:11.089511 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.084214    2569 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 12 17:03:11.089511 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.084218    2569 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 12 17:03:11.089511 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.084221    2569 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 12 17:03:11.089511 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.084223    2569 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 12 17:03:11.089511 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.084233    2569 flags.go:64] FLAG: --image-service-endpoint=""
Jul 12 17:03:11.089511 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.084236    2569 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 12 17:03:11.089511 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.084240    2569 flags.go:64] FLAG: --kube-api-burst="100"
Jul 12 17:03:11.090134 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.084243    2569 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 12 17:03:11.090134 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.084246    2569 flags.go:64] FLAG: --kube-api-qps="50"
Jul 12 17:03:11.090134 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.084249    2569 flags.go:64] FLAG: --kube-reserved=""
Jul 12 17:03:11.090134 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.084252    2569 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 12 17:03:11.090134 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.084255    2569 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 12 17:03:11.090134 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.084258    2569 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 12 17:03:11.090134 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.084261    2569 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 12 17:03:11.090134 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.084264    2569 flags.go:64] FLAG: --lock-file=""
Jul 12 17:03:11.090134 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.084267    2569 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 12 17:03:11.090134 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.084270    2569 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 12 17:03:11.090134 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.084273    2569 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 12 17:03:11.090134 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.084278    2569 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 12 17:03:11.090134 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.084281    2569 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 12 17:03:11.090134 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.084284    2569 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 12 17:03:11.090134 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.084287    2569 flags.go:64] FLAG: --logging-format="text"
Jul 12 17:03:11.090134 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.084290    2569 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 12 17:03:11.090134 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.084297    2569 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 12 17:03:11.090134 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.084300    2569 flags.go:64] FLAG: --manifest-url=""
Jul 12 17:03:11.090134 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.084303    2569 flags.go:64] FLAG: --manifest-url-header=""
Jul 12 17:03:11.090134 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.084307    2569 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 12 17:03:11.090134 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.084310    2569 flags.go:64] FLAG: --max-open-files="1000000"
Jul 12 17:03:11.090134 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.084314    2569 flags.go:64] FLAG: --max-pods="110"
Jul 12 17:03:11.090134 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.084317    2569 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 12 17:03:11.090134 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.084320    2569 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 12 17:03:11.090134 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.084323    2569 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 12 17:03:11.090755 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.084326    2569 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 12 17:03:11.090755 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.084329    2569 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 12 17:03:11.090755 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.084332    2569 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 12 17:03:11.090755 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.084335    2569 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 12 17:03:11.090755 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.084342    2569 flags.go:64] FLAG: --node-status-max-images="50"
Jul 12 17:03:11.090755 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.084345    2569 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 12 17:03:11.090755 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.084349    2569 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 12 17:03:11.090755 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.084352    2569 flags.go:64] FLAG: --pod-cidr=""
Jul 12 17:03:11.090755 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.084355    2569 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926"
Jul 12 17:03:11.090755 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.084361    2569 flags.go:64] FLAG: --pod-manifest-path=""
Jul 12 17:03:11.090755 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.084364    2569 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 12 17:03:11.090755 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.084367    2569 flags.go:64] FLAG: --pods-per-core="0"
Jul 12 17:03:11.090755 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.084370    2569 flags.go:64] FLAG: --port="10250"
Jul 12 17:03:11.090755 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.084373    2569 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 12 17:03:11.090755 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.084376    2569 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0752f716069e2bd22"
Jul 12 17:03:11.090755 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.084379    2569 flags.go:64] FLAG: --qos-reserved=""
Jul 12 17:03:11.090755 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.084384    2569 flags.go:64] FLAG: --read-only-port="10255"
Jul 12 17:03:11.090755 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.084387    2569 flags.go:64] FLAG: --register-node="true"
Jul 12 17:03:11.090755 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.084390    2569 flags.go:64] FLAG: --register-with-taints=""
Jul 12 17:03:11.090755 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.084394    2569 flags.go:64] FLAG: --registry-burst="10"
Jul 12 17:03:11.090755 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.084397    2569 flags.go:64] FLAG: --registry-qps="5"
Jul 12 17:03:11.090755 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.084399    2569 flags.go:64] FLAG: --reserved-cpus=""
Jul 12 17:03:11.090755 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.084402    2569 flags.go:64] FLAG: --reserved-memory=""
Jul 12 17:03:11.090755 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.084405    2569 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 12 17:03:11.090755 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.084409    2569 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 12 17:03:11.091776 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.084411    2569 flags.go:64] FLAG: --rotate-certificates="false"
Jul 12 17:03:11.091776 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.084414    2569 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 12 17:03:11.091776 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.084417    2569 flags.go:64] FLAG: --runonce="false"
Jul 12 17:03:11.091776 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.084420    2569 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 12 17:03:11.091776 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.084423    2569 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 12 17:03:11.091776 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.084426    2569 flags.go:64] FLAG: --seccomp-default="false"
Jul 12 17:03:11.091776 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.084429    2569 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 12 17:03:11.091776 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.084432    2569 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 12 17:03:11.091776 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.084435    2569 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 12 17:03:11.091776 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.084438    2569 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 12 17:03:11.091776 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.084441    2569 flags.go:64] FLAG: --storage-driver-password="root"
Jul 12 17:03:11.091776 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.084444    2569 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 12 17:03:11.091776 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.084447    2569 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 12 17:03:11.091776 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.084450    2569 flags.go:64] FLAG: --storage-driver-user="root"
Jul 12 17:03:11.091776 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.084454    2569 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 12 17:03:11.091776 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.084457    2569 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 12 17:03:11.091776 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.084460    2569 flags.go:64] FLAG: --system-cgroups=""
Jul 12 17:03:11.091776 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.084463    2569 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 12 17:03:11.091776 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.084468    2569 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 12 17:03:11.091776 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.084471    2569 flags.go:64] FLAG: --tls-cert-file=""
Jul 12 17:03:11.091776 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.084473    2569 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 12 17:03:11.091776 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.084478    2569 flags.go:64] FLAG: --tls-min-version=""
Jul 12 17:03:11.091776 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.084481    2569 flags.go:64] FLAG: --tls-private-key-file=""
Jul 12 17:03:11.091776 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.084486    2569 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 12 17:03:11.091776 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.084488    2569 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 12 17:03:11.092730 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.084491    2569 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 12 17:03:11.092730 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.084495    2569 flags.go:64] FLAG: --v="2"
Jul 12 17:03:11.092730 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.084498    2569 flags.go:64] FLAG: --version="false"
Jul 12 17:03:11.092730 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.084502    2569 flags.go:64] FLAG: --vmodule=""
Jul 12 17:03:11.092730 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.084505    2569 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 12 17:03:11.092730 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.084508    2569 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 12 17:03:11.092730 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.084617    2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 12 17:03:11.092730 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.084621    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 12 17:03:11.092730 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.084624    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 12 17:03:11.092730 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.084627    2569 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 12 17:03:11.092730 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.084629    2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 12 17:03:11.092730 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.084632    2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 12 17:03:11.092730 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.084634    2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 12 17:03:11.092730 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.084637    2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 12 17:03:11.092730 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.084640    2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 12 17:03:11.092730 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.084642    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 12 17:03:11.092730 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.084645    2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 12 17:03:11.092730 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.084648    2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 12 17:03:11.092730 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.084650    2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 12 17:03:11.092730 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.084653    2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 12 17:03:11.092730 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.084656    2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 12 17:03:11.093388 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.084660    2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 12 17:03:11.093388 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.084662    2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 12 17:03:11.093388 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.084665    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 12 17:03:11.093388 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.084668    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 12 17:03:11.093388 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.084670    2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 12 17:03:11.093388 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.084674    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 12 17:03:11.093388 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.084676    2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 12 17:03:11.093388 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.084679    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 12 17:03:11.093388 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.084682    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 12 17:03:11.093388 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.084686    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 12 17:03:11.093388 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.084688    2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 12 17:03:11.093388 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.084691    2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 12 17:03:11.093388 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.084694    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 12 17:03:11.093388 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.084696    2569 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 12 17:03:11.093388 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.084699    2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 12 17:03:11.093388 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.084701    2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 12 17:03:11.093388 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.084722    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 12 17:03:11.093388 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.084727    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 12 17:03:11.093388 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.084730    2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 12 17:03:11.093388 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.084733    2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 12 17:03:11.093937 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.084736    2569 feature_gate.go:328] unrecognized feature gate: Example2
Jul 12 17:03:11.093937 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.084738    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 12 17:03:11.093937 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.084741    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 12 17:03:11.093937 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.084743    2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 12 17:03:11.093937 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.084746    2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 12 17:03:11.093937 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.084749    2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 12 17:03:11.093937 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.084751    2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 12 17:03:11.093937 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.084755    2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 12 17:03:11.093937 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.084759    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 12 17:03:11.093937 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.084762    2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 12 17:03:11.093937 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.084765    2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 12 17:03:11.093937 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.084768    2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 12 17:03:11.093937 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.084772    2569 feature_gate.go:328] unrecognized feature gate: Example
Jul 12 17:03:11.093937 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.084774    2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 12 17:03:11.093937 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.084777    2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 12 17:03:11.093937 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.084780    2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 12 17:03:11.093937 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.084784    2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 12 17:03:11.093937 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.084787    2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 12 17:03:11.093937 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.084790    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 12 17:03:11.093937 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.084792    2569 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 12 17:03:11.094461 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.084795    2569 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 12 17:03:11.094461 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.084799    2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 12 17:03:11.094461 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.084801    2569 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 12 17:03:11.094461 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.084804    2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 12 17:03:11.094461 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.084806    2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 12 17:03:11.094461 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.084809    2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 12 17:03:11.094461 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.084812    2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 12 17:03:11.094461 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.084814    2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 12 17:03:11.094461 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.084817    2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 12 17:03:11.094461 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.084819    2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 12 17:03:11.094461 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.084822    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 12 17:03:11.094461 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.084824    2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 12 17:03:11.094461 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.084827    2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 12 17:03:11.094461 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.084829    2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 12 17:03:11.094461 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.084832    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 12 17:03:11.094461 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.084834    2569 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 12 17:03:11.094461 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.084837    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 12 17:03:11.094461 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.084840    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 12 17:03:11.094461 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.084842    2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 12 17:03:11.094950 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.084846    2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 12 17:03:11.094950 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.084849    2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 12 17:03:11.094950 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.084852    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 12 17:03:11.094950 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.084856    2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 12 17:03:11.094950 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.084859    2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 12 17:03:11.094950 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.084863    2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 12 17:03:11.094950 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.084866    2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 12 17:03:11.094950 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.084869    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 12 17:03:11.094950 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.084872    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 12 17:03:11.094950 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.084874    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 12 17:03:11.094950 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.084877    2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 12 17:03:11.094950 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.084880    2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 12 17:03:11.094950 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.084883    2569 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 12 17:03:11.094950 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.084885    2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 12 17:03:11.094950 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.084889    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 12 17:03:11.094950 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.084892    2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 12 17:03:11.094950 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.084895    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 12 17:03:11.094950 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.084897    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 12 17:03:11.094950 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.084899    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 12 17:03:11.095406 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.084902    2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 12 17:03:11.095406 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.084904    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 12 17:03:11.095406 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.084907    2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 12 17:03:11.095406 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.084910    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 12 17:03:11.095406 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.084912    2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 12 17:03:11.095406 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.084914    2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 12 17:03:11.095406 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.085793    2569 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 12 17:03:11.095406 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.094043    2569 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 12 17:03:11.095406 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.094057    2569 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 12 17:03:11.095406 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.094082    2569 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 12 17:03:11.095406 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.094088    2569 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 12 17:03:11.095406 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094142    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 12 17:03:11.095406 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094148    2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 12 17:03:11.095406 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094151    2569 feature_gate.go:328] unrecognized feature gate: Example
Jul 12 17:03:11.095406 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094155    2569 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 12 17:03:11.095785 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094158    2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 12 17:03:11.095785 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094162    2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 12 17:03:11.095785 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094164    2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 12 17:03:11.095785 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094167    2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 12 17:03:11.095785 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094170    2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 12 17:03:11.095785 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094172    2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 12 17:03:11.095785 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094175    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 12 17:03:11.095785 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094178    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 12 17:03:11.095785 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094180    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 12 17:03:11.095785 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094183    2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 12 17:03:11.095785 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094185    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 12 17:03:11.095785 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094188    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 12 17:03:11.095785 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094191    2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 12 17:03:11.095785 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094193    2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 12 17:03:11.095785 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094196    2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 12 17:03:11.095785 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094198    2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 12 17:03:11.095785 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094201    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 12 17:03:11.095785 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094204    2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 12 17:03:11.095785 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094207    2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 12 17:03:11.095785 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094209    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 12 17:03:11.096272 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094212    2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 12 17:03:11.096272 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094215    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 12 17:03:11.096272 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094218    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 12 17:03:11.096272 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094220    2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 12 17:03:11.096272 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094223    2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 12 17:03:11.096272 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094226    2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 12 17:03:11.096272 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094228    2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 12 17:03:11.096272 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094231    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 12 17:03:11.096272 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094234    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 12 17:03:11.096272 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094237    2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 12 17:03:11.096272 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094239    2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 12 17:03:11.096272 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094242    2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 12 17:03:11.096272 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094245    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 12 17:03:11.096272 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094247    2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 12 17:03:11.096272 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094250    2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 12 17:03:11.096272 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094252    2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 12 17:03:11.096272 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094255    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 12 17:03:11.096272 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094258    2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 12 17:03:11.096272 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094260    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 12 17:03:11.096272 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094263    2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 12 17:03:11.096776 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094265    2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 12 17:03:11.096776 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094268    2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 12 17:03:11.096776 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094271    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 12 17:03:11.096776 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094273    2569 feature_gate.go:328] unrecognized feature gate: Example2
Jul 12 17:03:11.096776 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094275    2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 12 17:03:11.096776 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094278    2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 12 17:03:11.096776 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094281    2569 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 12 17:03:11.096776 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094283    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 12 17:03:11.096776 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094286    2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 12 17:03:11.096776 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094288    2569 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 12 17:03:11.096776 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094291    2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 12 17:03:11.096776 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094294    2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 12 17:03:11.096776 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094296    2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 12 17:03:11.096776 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094299    2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 12 17:03:11.096776 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094302    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 12 17:03:11.096776 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094305    2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 12 17:03:11.096776 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094308    2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 12 17:03:11.096776 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094311    2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 12 17:03:11.096776 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094313    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 12 17:03:11.096776 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094316    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 12 17:03:11.097273 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094318    2569 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 12 17:03:11.097273 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094321    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 12 17:03:11.097273 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094323    2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 12 17:03:11.097273 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094326    2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 12 17:03:11.097273 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094328    2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 12 17:03:11.097273 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094331    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 12 17:03:11.097273 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094333    2569 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 12 17:03:11.097273 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094336    2569 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 12 17:03:11.097273 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094338    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 12 17:03:11.097273 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094341    2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 12 17:03:11.097273 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094343    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 12 17:03:11.097273 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094346    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 12 17:03:11.097273 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094348    2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 12 17:03:11.097273 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094351    2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 12 17:03:11.097273 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094353    2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 12 17:03:11.097273 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094356    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 12 17:03:11.097273 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094359    2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 12 17:03:11.097273 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094362    2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 12 17:03:11.097273 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094364    2569 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 12 17:03:11.097273 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094367    2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 12 17:03:11.097780 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094369    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 12 17:03:11.097780 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094373    2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 12 17:03:11.097780 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094377    2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 12 17:03:11.097780 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094380    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 12 17:03:11.097780 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094383    2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 12 17:03:11.097780 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094388    2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 12 17:03:11.097780 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094392    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 12 17:03:11.097780 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094395    2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 12 17:03:11.097780 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094399    2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 12 17:03:11.097780 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094402    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 12 17:03:11.097780 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094405    2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 12 17:03:11.097780 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094408    2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 12 17:03:11.097780 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094411    2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 12 17:03:11.097780 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094414    2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 12 17:03:11.097780 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094417    2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 12 17:03:11.098153 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.094422    2569 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 12 17:03:11.098153 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094527    2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 12 17:03:11.098153 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094532    2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 12 17:03:11.098153 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094535    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 12 17:03:11.098153 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094537    2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 12 17:03:11.098153 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094540    2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 12 17:03:11.098153 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094543    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 12 17:03:11.098153 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094546    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 12 17:03:11.098153 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094550    2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 12 17:03:11.098153 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094554    2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 12 17:03:11.098153 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094557    2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 12 17:03:11.098153 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094560    2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 12 17:03:11.098153 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094563    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 12 17:03:11.098153 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094567    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 12 17:03:11.098153 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094569    2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 12 17:03:11.098575 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094572    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 12 17:03:11.098575 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094574    2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 12 17:03:11.098575 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094577    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 12 17:03:11.098575 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094580    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 12 17:03:11.098575 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094582    2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 12 17:03:11.098575 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094585    2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 12 17:03:11.098575 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094587    2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 12 17:03:11.098575 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094590    2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 12 17:03:11.098575 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094593    2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 12 17:03:11.098575 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094596    2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 12 17:03:11.098575 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094599    2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 12 17:03:11.098575 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094601    2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 12 17:03:11.098575 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094604    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 12 17:03:11.098575 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094607    2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 12 17:03:11.098575 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094609    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 12 17:03:11.098575 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094611    2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 12 17:03:11.098575 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094614    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 12 17:03:11.098575 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094617    2569 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 12 17:03:11.098575 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094619    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 12 17:03:11.098575 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094622    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 12 17:03:11.099189 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094624    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 12 17:03:11.099189 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094627    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 12 17:03:11.099189 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094629    2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 12 17:03:11.099189 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094632    2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 12 17:03:11.099189 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094634    2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 12 17:03:11.099189 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094637    2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 12 17:03:11.099189 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094639    2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 12 17:03:11.099189 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094642    2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 12 17:03:11.099189 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094644    2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 12 17:03:11.099189 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094647    2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 12 17:03:11.099189 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094649    2569 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 12 17:03:11.099189 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094656    2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 12 17:03:11.099189 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094658    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 12 17:03:11.099189 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094661    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 12 17:03:11.099189 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094664    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 12 17:03:11.099189 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094666    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 12 17:03:11.099189 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094670    2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 12 17:03:11.099189 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094674    2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 12 17:03:11.099189 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094676    2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 12 17:03:11.099736 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094679    2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 12 17:03:11.099736 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094682    2569 feature_gate.go:328] unrecognized feature gate: Example
Jul 12 17:03:11.099736 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094685    2569 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 12 17:03:11.099736 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094688    2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 12 17:03:11.099736 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094690    2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 12 17:03:11.099736 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094693    2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 12 17:03:11.099736 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094696    2569 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 12 17:03:11.099736 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094698    2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 12 17:03:11.099736 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094700    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 12 17:03:11.099736 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094716    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 12 17:03:11.099736 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094719    2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 12 17:03:11.099736 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094722    2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 12 17:03:11.099736 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094725    2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 12 17:03:11.099736 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094728    2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 12 17:03:11.099736 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094730    2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 12 17:03:11.099736 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094733    2569 feature_gate.go:328] unrecognized feature gate: Example2
Jul 12 17:03:11.099736 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094735    2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 12 17:03:11.099736 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094738    2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 12 17:03:11.099736 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094741    2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 12 17:03:11.099736 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094743    2569 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 12 17:03:11.099736 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094746    2569 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 12 17:03:11.100299 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094749    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 12 17:03:11.100299 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094751    2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 12 17:03:11.100299 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094754    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 12 17:03:11.100299 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094756    2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 12 17:03:11.100299 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094759    2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 12 17:03:11.100299 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094762    2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 12 17:03:11.100299 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094765    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 12 17:03:11.100299 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094768    2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 12 17:03:11.100299 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094770    2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 12 17:03:11.100299 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094773    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 12 17:03:11.100299 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094776    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 12 17:03:11.100299 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094779    2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 12 17:03:11.100299 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094782    2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 12 17:03:11.100299 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094784    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 12 17:03:11.100299 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094787    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 12 17:03:11.100299 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094791    2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 12 17:03:11.100299 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094794    2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 12 17:03:11.100299 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094797    2569 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 12 17:03:11.100299 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094800    2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 12 17:03:11.101111 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094803    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 12 17:03:11.101111 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094806    2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 12 17:03:11.101111 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094809    2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 12 17:03:11.101111 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094812    2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 12 17:03:11.101111 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094814    2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 12 17:03:11.101111 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:11.094817    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 12 17:03:11.101111 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.094822    2569 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 12 17:03:11.101111 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.094936    2569 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 12 17:03:11.101111 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.097064    2569 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 12 17:03:11.101111 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.098381    2569 server.go:1019] "Starting client certificate rotation"
Jul 12 17:03:11.101111 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.098500    2569 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 12 17:03:11.101111 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.098546    2569 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 12 17:03:11.101111 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.100644    2569 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 12 17:03:11.102641 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.102601    2569 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 12 17:03:11.115260 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.115236    2569 log.go:25] "Validated CRI v1 runtime API"
Jul 12 17:03:11.117176 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.117160    2569 log.go:25] "Validated CRI v1 image API"
Jul 12 17:03:11.117985 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.117972    2569 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 12 17:03:11.120572 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.120548    2569 fs.go:135] Filesystem UUIDs: map[2de57d9c-cfae-472d-9603-13b84dec298e:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 86d957c7-77a4-41dd-9eb5-b3cc948b4bad:/dev/nvme0n1p3]
Jul 12 17:03:11.120661 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.120568    2569 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 12 17:03:11.125123 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.125104    2569 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 12 17:03:11.126484 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.126384    2569 manager.go:216] Machine: {Timestamp:2026-07-12 17:03:11.125907204 +0000 UTC m=+0.318025520 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3107130 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2fbeed4af180db5e2e54d08977d190 SystemUUID:ec2fbeed-4af1-80db-5e2e-54d08977d190 BootID:b28f51f2-60b0-40de-8090-63821c51168e Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:71:17:b7:f0:fb Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:71:17:b7:f0:fb Speed:0 Mtu:9001} {Name:ovs-system MacAddress:26:8e:43:eb:a5:56 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 12 17:03:11.126484 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.126478    2569 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 12 17:03:11.126594 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.126567    2569 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 12 17:03:11.130250 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.130227    2569 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 12 17:03:11.130375 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.130253    2569 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-94.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 12 17:03:11.130931 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.130921    2569 topology_manager.go:138] "Creating topology manager with none policy"
Jul 12 17:03:11.130961 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.130933    2569 container_manager_linux.go:309] "Creating device plugin manager"
Jul 12 17:03:11.130961 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.130940    2569 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 12 17:03:11.130961 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.130950    2569 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 12 17:03:11.132166 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.132156    2569 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 12 17:03:11.134901 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.134890    2569 state_mem.go:36] "Initialized new in-memory state store"
Jul 12 17:03:11.135007 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.134997    2569 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 12 17:03:11.135050 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.135024    2569 kubelet.go:486] "Attempting to sync node with API server"
Jul 12 17:03:11.135050 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.135032    2569 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 12 17:03:11.135050 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.135045    2569 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 12 17:03:11.135175 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.135054    2569 kubelet.go:398] "Adding apiserver pod source"
Jul 12 17:03:11.135175 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.135066    2569 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 12 17:03:11.135451 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.135420    2569 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 12 17:03:11.135498 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.135465    2569 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 12 17:03:11.137427 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.137406    2569 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 12 17:03:11.139277 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.139263    2569 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 12 17:03:11.139333 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.139294    2569 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 12 17:03:11.139434 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.139424    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 12 17:03:11.139463 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.139438    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 12 17:03:11.139463 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.139444    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 12 17:03:11.139463 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.139449    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 12 17:03:11.139463 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.139457    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 12 17:03:11.139463 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.139461    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 12 17:03:11.140002 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.139993    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 12 17:03:11.140002 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.140002    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 12 17:03:11.140062 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.140009    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 12 17:03:11.140062 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.140014    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 12 17:03:11.140062 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.140027    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 12 17:03:11.140062 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.140035    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 12 17:03:11.140836 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.140827    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 12 17:03:11.140836 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.140835    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 12 17:03:11.143144 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.143127    2569 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-141-94.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 12 17:03:11.143144 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:03:11.143126    2569 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 12 17:03:11.143246 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:03:11.143166    2569 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-141-94.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 12 17:03:11.143883 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.143872    2569 server.go:1268] "Started kubelet"
Jul 12 17:03:11.144024 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.143963    2569 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 12 17:03:11.144081 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.144044    2569 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 12 17:03:11.144215 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.144188    2569 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 12 17:03:11.144728 ip-10-0-141-94 systemd[1]: Started Kubernetes Kubelet.
Jul 12 17:03:11.145157 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.145135    2569 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 12 17:03:11.145872 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.145854    2569 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-v9fzp"
Jul 12 17:03:11.145957 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.145946    2569 server.go:310] "Adding debug handlers to kubelet server"
Jul 12 17:03:11.149282 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:03:11.148471    2569 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-94.ec2.internal.18c19994a68c45ca  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-94.ec2.internal,UID:ip-10-0-141-94.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-141-94.ec2.internal,},FirstTimestamp:2026-07-12 17:03:11.14384737 +0000 UTC m=+0.335965687,LastTimestamp:2026-07-12 17:03:11.14384737 +0000 UTC m=+0.335965687,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-94.ec2.internal,}"
Jul 12 17:03:11.149854 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:03:11.149833    2569 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 12 17:03:11.153640 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.153626    2569 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-v9fzp"
Jul 12 17:03:11.161065 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.161050    2569 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 12 17:03:11.161065 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.161059    2569 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 12 17:03:11.161210 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:03:11.161182    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-94.ec2.internal\" not found"
Jul 12 17:03:11.161975 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.161930    2569 factory.go:55] Registering systemd factory
Jul 12 17:03:11.161975 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.161948    2569 factory.go:223] Registration of the systemd container factory successfully
Jul 12 17:03:11.162144 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.162131    2569 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 12 17:03:11.162223 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.162144    2569 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 12 17:03:11.162223 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.162184    2569 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 12 17:03:11.162223 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.162151    2569 factory.go:153] Registering CRI-O factory
Jul 12 17:03:11.162541 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.162525    2569 reconstruct.go:97] "Volume reconstruction finished"
Jul 12 17:03:11.162608 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.162542    2569 reconciler.go:29] "Reconciler: start to sync state"
Jul 12 17:03:11.162608 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.162531    2569 factory.go:223] Registration of the crio container factory successfully
Jul 12 17:03:11.162865 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.162848    2569 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 12 17:03:11.162935 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:03:11.162870    2569 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-141-94.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 12 17:03:11.163177 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:03:11.163152    2569 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 12 17:03:11.164104 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.164086    2569 factory.go:103] Registering Raw factory
Jul 12 17:03:11.164308 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.164298    2569 manager.go:1196] Started watching for new ooms in manager
Jul 12 17:03:11.164953 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.164940    2569 manager.go:349] Starting recovery of all containers
Jul 12 17:03:11.174647 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.174630    2569 manager.go:354] Recovery completed
Jul 12 17:03:11.178425 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.178412    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 12 17:03:11.180541 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.180525    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-94.ec2.internal" event="NodeHasSufficientMemory"
Jul 12 17:03:11.180597 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.180553    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-94.ec2.internal" event="NodeHasNoDiskPressure"
Jul 12 17:03:11.180597 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.180566    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-94.ec2.internal" event="NodeHasSufficientPID"
Jul 12 17:03:11.181075 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.181061    2569 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 12 17:03:11.181075 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.181074    2569 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 12 17:03:11.181172 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.181076    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 12 17:03:11.181172 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.181089    2569 state_mem.go:36] "Initialized new in-memory state store"
Jul 12 17:03:11.181864 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.181849    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-94.ec2.internal" event="NodeHasSufficientMemory"
Jul 12 17:03:11.181924 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.181874    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-94.ec2.internal" event="NodeHasNoDiskPressure"
Jul 12 17:03:11.181924 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.181887    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-94.ec2.internal" event="NodeHasSufficientPID"
Jul 12 17:03:11.182398 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.182387    2569 policy_none.go:49] "None policy: Start"
Jul 12 17:03:11.182429 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.182402    2569 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 12 17:03:11.182429 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.182411    2569 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 12 17:03:11.183422 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.183412    2569 policy_none.go:47] "Start"
Jul 12 17:03:11.218250 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.218234    2569 manager.go:337] "Starting Device Plugin manager"
Jul 12 17:03:11.218315 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:03:11.218268    2569 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 12 17:03:11.218315 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.218277    2569 server.go:87] "Starting device plugin registration server"
Jul 12 17:03:11.218504 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.218489    2569 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 12 17:03:11.218584 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.218505    2569 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 12 17:03:11.218584 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.218576    2569 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 12 17:03:11.218693 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.218652    2569 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 12 17:03:11.218693 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.218661    2569 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 12 17:03:11.219216 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:03:11.219196    2569 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 12 17:03:11.219277 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:03:11.219239    2569 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-94.ec2.internal\" not found"
Jul 12 17:03:11.300165 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.300113    2569 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 12 17:03:11.301255 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.301235    2569 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 12 17:03:11.301333 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.301271    2569 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 12 17:03:11.301333 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.301291    2569 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 12 17:03:11.301333 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:03:11.301326    2569 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 12 17:03:11.303654 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.303630    2569 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 12 17:03:11.319402 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.319377    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 12 17:03:11.320178 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.320161    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-94.ec2.internal" event="NodeHasSufficientMemory"
Jul 12 17:03:11.320263 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.320194    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-94.ec2.internal" event="NodeHasNoDiskPressure"
Jul 12 17:03:11.320263 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.320209    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-94.ec2.internal" event="NodeHasSufficientPID"
Jul 12 17:03:11.320263 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.320238    2569 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-141-94.ec2.internal"
Jul 12 17:03:11.328036 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:11.327995    2569 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-141-94.ec2.internal"
Jul 12 17:03:11.328036 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:03:11.328014    2569 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-141-94.ec2.internal\": node \"ip-10-0-141-94.ec2.internal\" not found"
Jul 12 17:03:11.346047 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:03:11.346024    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-94.ec2.internal\" not found"
Jul 12 17:03:12.474867 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:12.473961    2569 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-141-94.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-94.ec2.internal"]
Jul 12 17:03:12.474935 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:12.474906    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 12 17:03:12.475694 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:12.475675    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-94.ec2.internal" event="NodeHasSufficientMemory"
Jul 12 17:03:12.475785 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:12.475709    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-94.ec2.internal" event="NodeHasNoDiskPressure"
Jul 12 17:03:12.475785 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:12.475723    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-94.ec2.internal" event="NodeHasSufficientPID"
Jul 12 17:03:12.477028 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:12.477014    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 12 17:03:12.477137 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:12.477111    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-94.ec2.internal"
Jul 12 17:03:12.477204 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:12.477144    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 12 17:03:12.477710 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:12.477695    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-94.ec2.internal" event="NodeHasSufficientMemory"
Jul 12 17:03:12.477710 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:12.477707    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-94.ec2.internal" event="NodeHasSufficientMemory"
Jul 12 17:03:12.477828 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:12.477719    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-94.ec2.internal" event="NodeHasNoDiskPressure"
Jul 12 17:03:12.477828 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:12.477725    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-94.ec2.internal" event="NodeHasNoDiskPressure"
Jul 12 17:03:12.477828 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:12.477730    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-94.ec2.internal" event="NodeHasSufficientPID"
Jul 12 17:03:12.477828 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:12.477735    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-94.ec2.internal" event="NodeHasSufficientPID"
Jul 12 17:03:12.479022 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:12.479006    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-94.ec2.internal"
Jul 12 17:03:12.479115 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:12.479028    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 12 17:03:12.479620 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:12.479607    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-94.ec2.internal" event="NodeHasSufficientMemory"
Jul 12 17:03:12.479697 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:12.479632    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-94.ec2.internal" event="NodeHasNoDiskPressure"
Jul 12 17:03:12.479697 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:12.479646    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-94.ec2.internal" event="NodeHasSufficientPID"
Jul 12 17:03:12.500120 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:03:12.500096    2569 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-94.ec2.internal\" not found" node="ip-10-0-141-94.ec2.internal"
Jul 12 17:03:12.504570 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:03:12.504554    2569 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-94.ec2.internal\" not found" node="ip-10-0-141-94.ec2.internal"
Jul 12 17:03:12.518557 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:03:12.518540    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-94.ec2.internal\" not found"
Jul 12 17:03:12.535669 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:12.535647    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/64598ecc4535f6658a01b7cab4f3caeb-config\") pod \"kube-apiserver-proxy-ip-10-0-141-94.ec2.internal\" (UID: \"64598ecc4535f6658a01b7cab4f3caeb\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-94.ec2.internal"
Jul 12 17:03:12.535739 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:12.535673    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bf9d9cd767ec6f08a024f6a8e7fbc81d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-94.ec2.internal\" (UID: \"bf9d9cd767ec6f08a024f6a8e7fbc81d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-94.ec2.internal"
Jul 12 17:03:12.535739 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:12.535690    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bf9d9cd767ec6f08a024f6a8e7fbc81d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-94.ec2.internal\" (UID: \"bf9d9cd767ec6f08a024f6a8e7fbc81d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-94.ec2.internal"
Jul 12 17:03:12.619021 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:03:12.618991    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-94.ec2.internal\" not found"
Jul 12 17:03:12.636341 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:12.636315    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/64598ecc4535f6658a01b7cab4f3caeb-config\") pod \"kube-apiserver-proxy-ip-10-0-141-94.ec2.internal\" (UID: \"64598ecc4535f6658a01b7cab4f3caeb\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-94.ec2.internal"
Jul 12 17:03:12.636441 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:12.636347    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bf9d9cd767ec6f08a024f6a8e7fbc81d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-94.ec2.internal\" (UID: \"bf9d9cd767ec6f08a024f6a8e7fbc81d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-94.ec2.internal"
Jul 12 17:03:12.636441 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:12.636373    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bf9d9cd767ec6f08a024f6a8e7fbc81d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-94.ec2.internal\" (UID: \"bf9d9cd767ec6f08a024f6a8e7fbc81d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-94.ec2.internal"
Jul 12 17:03:12.636441 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:12.636418    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/64598ecc4535f6658a01b7cab4f3caeb-config\") pod \"kube-apiserver-proxy-ip-10-0-141-94.ec2.internal\" (UID: \"64598ecc4535f6658a01b7cab4f3caeb\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-94.ec2.internal"
Jul 12 17:03:12.636576 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:12.636421    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bf9d9cd767ec6f08a024f6a8e7fbc81d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-94.ec2.internal\" (UID: \"bf9d9cd767ec6f08a024f6a8e7fbc81d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-94.ec2.internal"
Jul 12 17:03:12.636576 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:12.636421    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bf9d9cd767ec6f08a024f6a8e7fbc81d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-94.ec2.internal\" (UID: \"bf9d9cd767ec6f08a024f6a8e7fbc81d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-94.ec2.internal"
Jul 12 17:03:12.719712 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:03:12.719653    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-94.ec2.internal\" not found"
Jul 12 17:03:12.802189 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:12.802160    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-94.ec2.internal"
Jul 12 17:03:12.808106 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:12.808093    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-94.ec2.internal"
Jul 12 17:03:12.819836 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:03:12.819817    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-94.ec2.internal\" not found"
Jul 12 17:03:12.920437 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:03:12.920403    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-94.ec2.internal\" not found"
Jul 12 17:03:13.020985 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:03:13.020936    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-94.ec2.internal\" not found"
Jul 12 17:03:13.109538 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.109502    2569 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 12 17:03:13.133576 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.133557    2569 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-94.ec2.internal"
Jul 12 17:03:13.143458 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.143441    2569 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 12 17:03:13.145097 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.145085    2569 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-94.ec2.internal"
Jul 12 17:03:13.155118 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.155101    2569 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 12 17:03:13.169192 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.169172    2569 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 12 17:03:13.169310 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.169292    2569 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 12 17:03:13.169357 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.169308    2569 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 12 17:03:13.198820 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.198799    2569 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 12 17:03:13.207366 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.207349    2569 apiserver.go:52] "Watching apiserver"
Jul 12 17:03:13.213971 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.213952    2569 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 12 17:03:13.214317 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.214298    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-zlfcq","openshift-image-registry/node-ca-gpzg5","openshift-multus/multus-xsskc","openshift-network-operator/iptables-alerter-9bc6r","openshift-ovn-kubernetes/ovnkube-node-9xbrj","kube-system/global-pull-secret-syncer-mkw2b","kube-system/konnectivity-agent-kjdjh","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-smtm5","openshift-cluster-node-tuning-operator/tuned-nvmct","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-94.ec2.internal","openshift-multus/multus-additional-cni-plugins-ltcld","openshift-multus/network-metrics-daemon-h442x","openshift-network-diagnostics/network-check-target-gxtt8","kube-system/kube-apiserver-proxy-ip-10-0-141-94.ec2.internal"]
Jul 12 17:03:13.215733 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.215717    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-kjdjh"
Jul 12 17:03:13.216830 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.216813    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gxtt8"
Jul 12 17:03:13.216948 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:03:13.216867    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gxtt8" podUID="259ec8ed-10bd-4fae-8f7f-97409bd4f54b"
Jul 12 17:03:13.217979 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.217962    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 12 17:03:13.218075 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.217967    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-hjnhd\""
Jul 12 17:03:13.218404 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.218383    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-xsskc"
Jul 12 17:03:13.219008 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.218993    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 12 17:03:13.219928 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.219916    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-9bc6r"
Jul 12 17:03:13.220299 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.220283    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 12 17:03:13.220501 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.220487    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 12 17:03:13.220611 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.220592    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 12 17:03:13.220672 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.220650    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 12 17:03:13.220875 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.220859    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-5f75z\""
Jul 12 17:03:13.221100 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.221087    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-9xbrj"
Jul 12 17:03:13.221700 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.221684    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 12 17:03:13.222188 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.222170    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-qlwk5\""
Jul 12 17:03:13.222277 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.222252    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 12 17:03:13.222358 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.222340    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 12 17:03:13.222414 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.222363    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mkw2b"
Jul 12 17:03:13.222464 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:03:13.222419    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mkw2b" podUID="577e3a93-0560-4099-bddb-dd55563d002d"
Jul 12 17:03:13.223152 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.223138    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-cxbn5\""
Jul 12 17:03:13.223220 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.223152    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 12 17:03:13.223390 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.223376    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 12 17:03:13.223593 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.223574    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-nvmct"
Jul 12 17:03:13.224284 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.224266    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 12 17:03:13.224424 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.224407    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 12 17:03:13.224610 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.224575    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 12 17:03:13.224965 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.224729    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 12 17:03:13.225219 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.225186    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-zlfcq"
Jul 12 17:03:13.225927 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.225904    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 12 17:03:13.226148 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.226127    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-shwc4\""
Jul 12 17:03:13.226572 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.226551    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 12 17:03:13.228055 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.227995    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-ts6sj\""
Jul 12 17:03:13.228175 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.228149    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 12 17:03:13.228401 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.228125    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-gpzg5"
Jul 12 17:03:13.229085 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.229066    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 12 17:03:13.230423 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.230406    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-smtm5"
Jul 12 17:03:13.230893 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.230879    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 12 17:03:13.231559 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.231157    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 12 17:03:13.231559 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.231216    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-6bhd6\""
Jul 12 17:03:13.231559 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.231264    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 12 17:03:13.231782 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.231764    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-ltcld"
Jul 12 17:03:13.232322 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.232304    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 12 17:03:13.232433 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.232319    2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-11 16:58:12 +0000 UTC" deadline="2028-01-19 02:38:49.597353987 +0000 UTC"
Jul 12 17:03:13.232494 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.232445    2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13329h35m36.364995197s"
Jul 12 17:03:13.232910 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.232890    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 12 17:03:13.232997 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.232921    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-ssf9h\""
Jul 12 17:03:13.232997 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.232934    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 12 17:03:13.233126 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.233032    2569 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 12 17:03:13.233268 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.233249    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h442x"
Jul 12 17:03:13.233367 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:03:13.233303    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h442x" podUID="f1d6c71b-d9fb-4f2a-9a81-5e8f0f8a6c13"
Jul 12 17:03:13.234079 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.234056    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 12 17:03:13.234182 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.234099    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 12 17:03:13.234240 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.234223    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-nxh5n\""
Jul 12 17:03:13.234913 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.234897    2569 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 12 17:03:13.239459 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.239440    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/59055e87-8c03-4225-97e1-62114754af67-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ltcld\" (UID: \"59055e87-8c03-4225-97e1-62114754af67\") " pod="openshift-multus/multus-additional-cni-plugins-ltcld"
Jul 12 17:03:13.239565 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.239465    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hk6x9\" (UniqueName: \"kubernetes.io/projected/fcdb5e54-4227-4aed-bcb8-b019fc5dfd86-kube-api-access-hk6x9\") pod \"node-resolver-zlfcq\" (UID: \"fcdb5e54-4227-4aed-bcb8-b019fc5dfd86\") " pod="openshift-dns/node-resolver-zlfcq"
Jul 12 17:03:13.239565 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.239483    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6b618491-4e81-4708-ab1b-2781d1df7c87-host-run-netns\") pod \"ovnkube-node-9xbrj\" (UID: \"6b618491-4e81-4708-ab1b-2781d1df7c87\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xbrj"
Jul 12 17:03:13.239565 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.239499    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2grnc\" (UniqueName: \"kubernetes.io/projected/6b618491-4e81-4708-ab1b-2781d1df7c87-kube-api-access-2grnc\") pod \"ovnkube-node-9xbrj\" (UID: \"6b618491-4e81-4708-ab1b-2781d1df7c87\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xbrj"
Jul 12 17:03:13.239565 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.239562    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/59055e87-8c03-4225-97e1-62114754af67-cni-binary-copy\") pod \"multus-additional-cni-plugins-ltcld\" (UID: \"59055e87-8c03-4225-97e1-62114754af67\") " pod="openshift-multus/multus-additional-cni-plugins-ltcld"
Jul 12 17:03:13.239711 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.239578    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8dc74bb2-e60b-49c3-9d2a-0bf12cdbc40a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-smtm5\" (UID: \"8dc74bb2-e60b-49c3-9d2a-0bf12cdbc40a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-smtm5"
Jul 12 17:03:13.239711 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.239611    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8dc74bb2-e60b-49c3-9d2a-0bf12cdbc40a-socket-dir\") pod \"aws-ebs-csi-driver-node-smtm5\" (UID: \"8dc74bb2-e60b-49c3-9d2a-0bf12cdbc40a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-smtm5"
Jul 12 17:03:13.239711 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.239635    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsprf\" (UniqueName: \"kubernetes.io/projected/8dc74bb2-e60b-49c3-9d2a-0bf12cdbc40a-kube-api-access-lsprf\") pod \"aws-ebs-csi-driver-node-smtm5\" (UID: \"8dc74bb2-e60b-49c3-9d2a-0bf12cdbc40a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-smtm5"
Jul 12 17:03:13.239711 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.239653    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ecab5f05-562e-47bc-9c5a-04893ab8a903-system-cni-dir\") pod \"multus-xsskc\" (UID: \"ecab5f05-562e-47bc-9c5a-04893ab8a903\") " pod="openshift-multus/multus-xsskc"
Jul 12 17:03:13.239711 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.239668    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ecab5f05-562e-47bc-9c5a-04893ab8a903-host-var-lib-cni-multus\") pod \"multus-xsskc\" (UID: \"ecab5f05-562e-47bc-9c5a-04893ab8a903\") " pod="openshift-multus/multus-xsskc"
Jul 12 17:03:13.239711 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.239702    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6b618491-4e81-4708-ab1b-2781d1df7c87-etc-openvswitch\") pod \"ovnkube-node-9xbrj\" (UID: \"6b618491-4e81-4708-ab1b-2781d1df7c87\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xbrj"
Jul 12 17:03:13.239909 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.239723    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6b618491-4e81-4708-ab1b-2781d1df7c87-log-socket\") pod \"ovnkube-node-9xbrj\" (UID: \"6b618491-4e81-4708-ab1b-2781d1df7c87\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xbrj"
Jul 12 17:03:13.239909 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.239761    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8bcfc1cb-b938-4719-a485-dc3acd069a11-etc-sysconfig\") pod \"tuned-nvmct\" (UID: \"8bcfc1cb-b938-4719-a485-dc3acd069a11\") " pod="openshift-cluster-node-tuning-operator/tuned-nvmct"
Jul 12 17:03:13.239909 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.239785    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbh7x\" (UniqueName: \"kubernetes.io/projected/c8fd63f2-13ee-4231-8ea3-a83dd90f365a-kube-api-access-jbh7x\") pod \"iptables-alerter-9bc6r\" (UID: \"c8fd63f2-13ee-4231-8ea3-a83dd90f365a\") " pod="openshift-network-operator/iptables-alerter-9bc6r"
Jul 12 17:03:13.239909 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.239824    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/577e3a93-0560-4099-bddb-dd55563d002d-kubelet-config\") pod \"global-pull-secret-syncer-mkw2b\" (UID: \"577e3a93-0560-4099-bddb-dd55563d002d\") " pod="kube-system/global-pull-secret-syncer-mkw2b"
Jul 12 17:03:13.239909 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.239847    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fcdb5e54-4227-4aed-bcb8-b019fc5dfd86-tmp-dir\") pod \"node-resolver-zlfcq\" (UID: \"fcdb5e54-4227-4aed-bcb8-b019fc5dfd86\") " pod="openshift-dns/node-resolver-zlfcq"
Jul 12 17:03:13.239909 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.239896    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ecab5f05-562e-47bc-9c5a-04893ab8a903-cnibin\") pod \"multus-xsskc\" (UID: \"ecab5f05-562e-47bc-9c5a-04893ab8a903\") " pod="openshift-multus/multus-xsskc"
Jul 12 17:03:13.240204 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.239928    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8bcfc1cb-b938-4719-a485-dc3acd069a11-var-lib-kubelet\") pod \"tuned-nvmct\" (UID: \"8bcfc1cb-b938-4719-a485-dc3acd069a11\") " pod="openshift-cluster-node-tuning-operator/tuned-nvmct"
Jul 12 17:03:13.240204 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.239961    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/098853ee-bad5-44d4-b2ac-bc4aa97b5d1c-host\") pod \"node-ca-gpzg5\" (UID: \"098853ee-bad5-44d4-b2ac-bc4aa97b5d1c\") " pod="openshift-image-registry/node-ca-gpzg5"
Jul 12 17:03:13.240204 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.239983    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/098853ee-bad5-44d4-b2ac-bc4aa97b5d1c-serviceca\") pod \"node-ca-gpzg5\" (UID: \"098853ee-bad5-44d4-b2ac-bc4aa97b5d1c\") " pod="openshift-image-registry/node-ca-gpzg5"
Jul 12 17:03:13.240204 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.240005    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6b618491-4e81-4708-ab1b-2781d1df7c87-systemd-units\") pod \"ovnkube-node-9xbrj\" (UID: \"6b618491-4e81-4708-ab1b-2781d1df7c87\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xbrj"
Jul 12 17:03:13.240204 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.240044    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6b618491-4e81-4708-ab1b-2781d1df7c87-var-lib-openvswitch\") pod \"ovnkube-node-9xbrj\" (UID: \"6b618491-4e81-4708-ab1b-2781d1df7c87\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xbrj"
Jul 12 17:03:13.240204 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.240066    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8dc74bb2-e60b-49c3-9d2a-0bf12cdbc40a-sys-fs\") pod \"aws-ebs-csi-driver-node-smtm5\" (UID: \"8dc74bb2-e60b-49c3-9d2a-0bf12cdbc40a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-smtm5"
Jul 12 17:03:13.240204 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.240089    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9gdk\" (UniqueName: \"kubernetes.io/projected/f1d6c71b-d9fb-4f2a-9a81-5e8f0f8a6c13-kube-api-access-x9gdk\") pod \"network-metrics-daemon-h442x\" (UID: \"f1d6c71b-d9fb-4f2a-9a81-5e8f0f8a6c13\") " pod="openshift-multus/network-metrics-daemon-h442x"
Jul 12 17:03:13.240204 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.240121    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ecab5f05-562e-47bc-9c5a-04893ab8a903-multus-cni-dir\") pod \"multus-xsskc\" (UID: \"ecab5f05-562e-47bc-9c5a-04893ab8a903\") " pod="openshift-multus/multus-xsskc"
Jul 12 17:03:13.240204 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.240148    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ecab5f05-562e-47bc-9c5a-04893ab8a903-multus-daemon-config\") pod \"multus-xsskc\" (UID: \"ecab5f05-562e-47bc-9c5a-04893ab8a903\") " pod="openshift-multus/multus-xsskc"
Jul 12 17:03:13.240204 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.240173    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6b618491-4e81-4708-ab1b-2781d1df7c87-run-systemd\") pod \"ovnkube-node-9xbrj\" (UID: \"6b618491-4e81-4708-ab1b-2781d1df7c87\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xbrj"
Jul 12 17:03:13.240204 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.240201    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6b618491-4e81-4708-ab1b-2781d1df7c87-host-cni-netd\") pod \"ovnkube-node-9xbrj\" (UID: \"6b618491-4e81-4708-ab1b-2781d1df7c87\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xbrj"
Jul 12 17:03:13.240674 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.240240    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/577e3a93-0560-4099-bddb-dd55563d002d-dbus\") pod \"global-pull-secret-syncer-mkw2b\" (UID: \"577e3a93-0560-4099-bddb-dd55563d002d\") " pod="kube-system/global-pull-secret-syncer-mkw2b"
Jul 12 17:03:13.240674 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.240259    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6b618491-4e81-4708-ab1b-2781d1df7c87-env-overrides\") pod \"ovnkube-node-9xbrj\" (UID: \"6b618491-4e81-4708-ab1b-2781d1df7c87\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xbrj"
Jul 12 17:03:13.240674 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.240273    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8bcfc1cb-b938-4719-a485-dc3acd069a11-etc-kubernetes\") pod \"tuned-nvmct\" (UID: \"8bcfc1cb-b938-4719-a485-dc3acd069a11\") " pod="openshift-cluster-node-tuning-operator/tuned-nvmct"
Jul 12 17:03:13.240674 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.240287    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8bcfc1cb-b938-4719-a485-dc3acd069a11-etc-sysctl-d\") pod \"tuned-nvmct\" (UID: \"8bcfc1cb-b938-4719-a485-dc3acd069a11\") " pod="openshift-cluster-node-tuning-operator/tuned-nvmct"
Jul 12 17:03:13.240674 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.240312    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mg5l\" (UniqueName: \"kubernetes.io/projected/8bcfc1cb-b938-4719-a485-dc3acd069a11-kube-api-access-2mg5l\") pod \"tuned-nvmct\" (UID: \"8bcfc1cb-b938-4719-a485-dc3acd069a11\") " pod="openshift-cluster-node-tuning-operator/tuned-nvmct"
Jul 12 17:03:13.240674 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.240331    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5598dd9a-714f-4e7d-9c59-a6e2f61c5a74-konnectivity-ca\") pod \"konnectivity-agent-kjdjh\" (UID: \"5598dd9a-714f-4e7d-9c59-a6e2f61c5a74\") " pod="kube-system/konnectivity-agent-kjdjh"
Jul 12 17:03:13.240674 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.240349    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6b618491-4e81-4708-ab1b-2781d1df7c87-host-cni-bin\") pod \"ovnkube-node-9xbrj\" (UID: \"6b618491-4e81-4708-ab1b-2781d1df7c87\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xbrj"
Jul 12 17:03:13.240674 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.240378    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ecab5f05-562e-47bc-9c5a-04893ab8a903-host-var-lib-kubelet\") pod \"multus-xsskc\" (UID: \"ecab5f05-562e-47bc-9c5a-04893ab8a903\") " pod="openshift-multus/multus-xsskc"
Jul 12 17:03:13.240674 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.240396    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6b618491-4e81-4708-ab1b-2781d1df7c87-ovnkube-config\") pod \"ovnkube-node-9xbrj\" (UID: \"6b618491-4e81-4708-ab1b-2781d1df7c87\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xbrj"
Jul 12 17:03:13.240674 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.240410    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ecab5f05-562e-47bc-9c5a-04893ab8a903-host-run-multus-certs\") pod \"multus-xsskc\" (UID: \"ecab5f05-562e-47bc-9c5a-04893ab8a903\") " pod="openshift-multus/multus-xsskc"
Jul 12 17:03:13.240674 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.240430    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ecab5f05-562e-47bc-9c5a-04893ab8a903-etc-kubernetes\") pod \"multus-xsskc\" (UID: \"ecab5f05-562e-47bc-9c5a-04893ab8a903\") " pod="openshift-multus/multus-xsskc"
Jul 12 17:03:13.240674 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.240446    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6b618491-4e81-4708-ab1b-2781d1df7c87-run-openvswitch\") pod \"ovnkube-node-9xbrj\" (UID: \"6b618491-4e81-4708-ab1b-2781d1df7c87\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xbrj"
Jul 12 17:03:13.240674 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.240474    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8bcfc1cb-b938-4719-a485-dc3acd069a11-tmp\") pod \"tuned-nvmct\" (UID: \"8bcfc1cb-b938-4719-a485-dc3acd069a11\") " pod="openshift-cluster-node-tuning-operator/tuned-nvmct"
Jul 12 17:03:13.240674 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.240497    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/59055e87-8c03-4225-97e1-62114754af67-system-cni-dir\") pod \"multus-additional-cni-plugins-ltcld\" (UID: \"59055e87-8c03-4225-97e1-62114754af67\") " pod="openshift-multus/multus-additional-cni-plugins-ltcld"
Jul 12 17:03:13.240674 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.240545    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8dc74bb2-e60b-49c3-9d2a-0bf12cdbc40a-device-dir\") pod \"aws-ebs-csi-driver-node-smtm5\" (UID: \"8dc74bb2-e60b-49c3-9d2a-0bf12cdbc40a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-smtm5"
Jul 12 17:03:13.240674 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.240568    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5598dd9a-714f-4e7d-9c59-a6e2f61c5a74-agent-certs\") pod \"konnectivity-agent-kjdjh\" (UID: \"5598dd9a-714f-4e7d-9c59-a6e2f61c5a74\") " pod="kube-system/konnectivity-agent-kjdjh"
Jul 12 17:03:13.240674 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.240596    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ecab5f05-562e-47bc-9c5a-04893ab8a903-hostroot\") pod \"multus-xsskc\" (UID: \"ecab5f05-562e-47bc-9c5a-04893ab8a903\") " pod="openshift-multus/multus-xsskc"
Jul 12 17:03:13.241374 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.240622    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8bcfc1cb-b938-4719-a485-dc3acd069a11-sys\") pod \"tuned-nvmct\" (UID: \"8bcfc1cb-b938-4719-a485-dc3acd069a11\") " pod="openshift-cluster-node-tuning-operator/tuned-nvmct"
Jul 12 17:03:13.241374 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.240639    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c8fd63f2-13ee-4231-8ea3-a83dd90f365a-host-slash\") pod \"iptables-alerter-9bc6r\" (UID: \"c8fd63f2-13ee-4231-8ea3-a83dd90f365a\") " pod="openshift-network-operator/iptables-alerter-9bc6r"
Jul 12 17:03:13.241374 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.240653    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1d6c71b-d9fb-4f2a-9a81-5e8f0f8a6c13-metrics-certs\") pod \"network-metrics-daemon-h442x\" (UID: \"f1d6c71b-d9fb-4f2a-9a81-5e8f0f8a6c13\") " pod="openshift-multus/network-metrics-daemon-h442x"
Jul 12 17:03:13.241374 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.240695    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xnbt\" (UniqueName: \"kubernetes.io/projected/ecab5f05-562e-47bc-9c5a-04893ab8a903-kube-api-access-6xnbt\") pod \"multus-xsskc\" (UID: \"ecab5f05-562e-47bc-9c5a-04893ab8a903\") " pod="openshift-multus/multus-xsskc"
Jul 12 17:03:13.241374 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.240719    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6b618491-4e81-4708-ab1b-2781d1df7c87-node-log\") pod \"ovnkube-node-9xbrj\" (UID: \"6b618491-4e81-4708-ab1b-2781d1df7c87\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xbrj"
Jul 12 17:03:13.241374 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.240742    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6b618491-4e81-4708-ab1b-2781d1df7c87-ovnkube-script-lib\") pod \"ovnkube-node-9xbrj\" (UID: \"6b618491-4e81-4708-ab1b-2781d1df7c87\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xbrj"
Jul 12 17:03:13.241374 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.240774    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8bcfc1cb-b938-4719-a485-dc3acd069a11-etc-modprobe-d\") pod \"tuned-nvmct\" (UID: \"8bcfc1cb-b938-4719-a485-dc3acd069a11\") " pod="openshift-cluster-node-tuning-operator/tuned-nvmct"
Jul 12 17:03:13.241374 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.240826    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/59055e87-8c03-4225-97e1-62114754af67-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ltcld\" (UID: \"59055e87-8c03-4225-97e1-62114754af67\") " pod="openshift-multus/multus-additional-cni-plugins-ltcld"
Jul 12 17:03:13.241374 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.240850    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fcdb5e54-4227-4aed-bcb8-b019fc5dfd86-hosts-file\") pod \"node-resolver-zlfcq\" (UID: \"fcdb5e54-4227-4aed-bcb8-b019fc5dfd86\") " pod="openshift-dns/node-resolver-zlfcq"
Jul 12 17:03:13.241374 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.240870    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8dc74bb2-e60b-49c3-9d2a-0bf12cdbc40a-etc-selinux\") pod \"aws-ebs-csi-driver-node-smtm5\" (UID: \"8dc74bb2-e60b-49c3-9d2a-0bf12cdbc40a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-smtm5"
Jul 12 17:03:13.241374 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.240898    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6sf9\" (UniqueName: \"kubernetes.io/projected/259ec8ed-10bd-4fae-8f7f-97409bd4f54b-kube-api-access-n6sf9\") pod \"network-check-target-gxtt8\" (UID: \"259ec8ed-10bd-4fae-8f7f-97409bd4f54b\") " pod="openshift-network-diagnostics/network-check-target-gxtt8"
Jul 12 17:03:13.241374 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.240920    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ecab5f05-562e-47bc-9c5a-04893ab8a903-host-var-lib-cni-bin\") pod \"multus-xsskc\" (UID: \"ecab5f05-562e-47bc-9c5a-04893ab8a903\") " pod="openshift-multus/multus-xsskc"
Jul 12 17:03:13.241374 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.240940    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6b618491-4e81-4708-ab1b-2781d1df7c87-host-kubelet\") pod \"ovnkube-node-9xbrj\" (UID: \"6b618491-4e81-4708-ab1b-2781d1df7c87\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xbrj"
Jul 12 17:03:13.241374 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.240960    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6b618491-4e81-4708-ab1b-2781d1df7c87-host-slash\") pod \"ovnkube-node-9xbrj\" (UID: \"6b618491-4e81-4708-ab1b-2781d1df7c87\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xbrj"
Jul 12 17:03:13.241374 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.240994    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8bcfc1cb-b938-4719-a485-dc3acd069a11-host\") pod \"tuned-nvmct\" (UID: \"8bcfc1cb-b938-4719-a485-dc3acd069a11\") " pod="openshift-cluster-node-tuning-operator/tuned-nvmct"
Jul 12 17:03:13.241374 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.241035    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/59055e87-8c03-4225-97e1-62114754af67-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ltcld\" (UID: \"59055e87-8c03-4225-97e1-62114754af67\") " pod="openshift-multus/multus-additional-cni-plugins-ltcld"
Jul 12 17:03:13.242082 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.241062    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c8fd63f2-13ee-4231-8ea3-a83dd90f365a-iptables-alerter-script\") pod \"iptables-alerter-9bc6r\" (UID: \"c8fd63f2-13ee-4231-8ea3-a83dd90f365a\") " pod="openshift-network-operator/iptables-alerter-9bc6r"
Jul 12 17:03:13.242082 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.241085    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8dc74bb2-e60b-49c3-9d2a-0bf12cdbc40a-registration-dir\") pod \"aws-ebs-csi-driver-node-smtm5\" (UID: \"8dc74bb2-e60b-49c3-9d2a-0bf12cdbc40a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-smtm5"
Jul 12 17:03:13.242082 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.241109    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ecab5f05-562e-47bc-9c5a-04893ab8a903-cni-binary-copy\") pod \"multus-xsskc\" (UID: \"ecab5f05-562e-47bc-9c5a-04893ab8a903\") " pod="openshift-multus/multus-xsskc"
Jul 12 17:03:13.242082 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.241130    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ecab5f05-562e-47bc-9c5a-04893ab8a903-multus-socket-dir-parent\") pod \"multus-xsskc\" (UID: \"ecab5f05-562e-47bc-9c5a-04893ab8a903\") " pod="openshift-multus/multus-xsskc"
Jul 12 17:03:13.242082 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.241160    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ecab5f05-562e-47bc-9c5a-04893ab8a903-host-run-k8s-cni-cncf-io\") pod \"multus-xsskc\" (UID: \"ecab5f05-562e-47bc-9c5a-04893ab8a903\") " pod="openshift-multus/multus-xsskc"
Jul 12 17:03:13.242082 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.241184    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b618491-4e81-4708-ab1b-2781d1df7c87-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9xbrj\" (UID: \"6b618491-4e81-4708-ab1b-2781d1df7c87\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xbrj"
Jul 12 17:03:13.242082 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.241228    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8bcfc1cb-b938-4719-a485-dc3acd069a11-lib-modules\") pod \"tuned-nvmct\" (UID: \"8bcfc1cb-b938-4719-a485-dc3acd069a11\") " pod="openshift-cluster-node-tuning-operator/tuned-nvmct"
Jul 12 17:03:13.242082 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.241251    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ecab5f05-562e-47bc-9c5a-04893ab8a903-os-release\") pod \"multus-xsskc\" (UID: \"ecab5f05-562e-47bc-9c5a-04893ab8a903\") " pod="openshift-multus/multus-xsskc"
Jul 12 17:03:13.242082 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.241272    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ecab5f05-562e-47bc-9c5a-04893ab8a903-host-run-netns\") pod \"multus-xsskc\" (UID: \"ecab5f05-562e-47bc-9c5a-04893ab8a903\") " pod="openshift-multus/multus-xsskc"
Jul 12 17:03:13.242082 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.241309    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ecab5f05-562e-47bc-9c5a-04893ab8a903-multus-conf-dir\") pod \"multus-xsskc\" (UID: \"ecab5f05-562e-47bc-9c5a-04893ab8a903\") " pod="openshift-multus/multus-xsskc"
Jul 12 17:03:13.242082 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.241332    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6b618491-4e81-4708-ab1b-2781d1df7c87-ovn-node-metrics-cert\") pod \"ovnkube-node-9xbrj\" (UID: \"6b618491-4e81-4708-ab1b-2781d1df7c87\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xbrj"
Jul 12 17:03:13.242082 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.241355    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8bcfc1cb-b938-4719-a485-dc3acd069a11-etc-systemd\") pod \"tuned-nvmct\" (UID: \"8bcfc1cb-b938-4719-a485-dc3acd069a11\") " pod="openshift-cluster-node-tuning-operator/tuned-nvmct"
Jul 12 17:03:13.242082 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.241393    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/577e3a93-0560-4099-bddb-dd55563d002d-original-pull-secret\") pod \"global-pull-secret-syncer-mkw2b\" (UID: \"577e3a93-0560-4099-bddb-dd55563d002d\") " pod="kube-system/global-pull-secret-syncer-mkw2b"
Jul 12 17:03:13.242082 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.241442    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9vfd\" (UniqueName: \"kubernetes.io/projected/098853ee-bad5-44d4-b2ac-bc4aa97b5d1c-kube-api-access-t9vfd\") pod \"node-ca-gpzg5\" (UID: \"098853ee-bad5-44d4-b2ac-bc4aa97b5d1c\") " pod="openshift-image-registry/node-ca-gpzg5"
Jul 12 17:03:13.242082 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.241489    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8bcfc1cb-b938-4719-a485-dc3acd069a11-run\") pod \"tuned-nvmct\" (UID: \"8bcfc1cb-b938-4719-a485-dc3acd069a11\") " pod="openshift-cluster-node-tuning-operator/tuned-nvmct"
Jul 12 17:03:13.242082 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.241533    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/59055e87-8c03-4225-97e1-62114754af67-cnibin\") pod \"multus-additional-cni-plugins-ltcld\" (UID: \"59055e87-8c03-4225-97e1-62114754af67\") " pod="openshift-multus/multus-additional-cni-plugins-ltcld"
Jul 12 17:03:13.242942 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.241562    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8htkx\" (UniqueName: \"kubernetes.io/projected/59055e87-8c03-4225-97e1-62114754af67-kube-api-access-8htkx\") pod \"multus-additional-cni-plugins-ltcld\" (UID: \"59055e87-8c03-4225-97e1-62114754af67\") " pod="openshift-multus/multus-additional-cni-plugins-ltcld"
Jul 12 17:03:13.242942 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.241594    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6b618491-4e81-4708-ab1b-2781d1df7c87-run-ovn\") pod \"ovnkube-node-9xbrj\" (UID: \"6b618491-4e81-4708-ab1b-2781d1df7c87\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xbrj"
Jul 12 17:03:13.242942 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.241617    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b618491-4e81-4708-ab1b-2781d1df7c87-host-run-ovn-kubernetes\") pod \"ovnkube-node-9xbrj\" (UID: \"6b618491-4e81-4708-ab1b-2781d1df7c87\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xbrj"
Jul 12 17:03:13.242942 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.241650    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8bcfc1cb-b938-4719-a485-dc3acd069a11-etc-sysctl-conf\") pod \"tuned-nvmct\" (UID: \"8bcfc1cb-b938-4719-a485-dc3acd069a11\") " pod="openshift-cluster-node-tuning-operator/tuned-nvmct"
Jul 12 17:03:13.242942 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.241687    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8bcfc1cb-b938-4719-a485-dc3acd069a11-etc-tuned\") pod \"tuned-nvmct\" (UID: \"8bcfc1cb-b938-4719-a485-dc3acd069a11\") " pod="openshift-cluster-node-tuning-operator/tuned-nvmct"
Jul 12 17:03:13.242942 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.241710    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/59055e87-8c03-4225-97e1-62114754af67-os-release\") pod \"multus-additional-cni-plugins-ltcld\" (UID: \"59055e87-8c03-4225-97e1-62114754af67\") " pod="openshift-multus/multus-additional-cni-plugins-ltcld"
Jul 12 17:03:13.246923 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.246896    2569 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 12 17:03:13.278117 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.278095    2569 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-w8xht"
Jul 12 17:03:13.285915 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.285899    2569 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-w8xht"
Jul 12 17:03:13.327000 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:13.326958    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64598ecc4535f6658a01b7cab4f3caeb.slice/crio-97756e8b166fb30026f6b43a0480bc22549a1a1ca1c721740371aea85631393f WatchSource:0}: Error finding container 97756e8b166fb30026f6b43a0480bc22549a1a1ca1c721740371aea85631393f: Status 404 returned error can't find the container with id 97756e8b166fb30026f6b43a0480bc22549a1a1ca1c721740371aea85631393f
Jul 12 17:03:13.327197 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:13.327180    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf9d9cd767ec6f08a024f6a8e7fbc81d.slice/crio-74130723b4eebe0c2c25bd5a181e11a0f3bcffec218791490249c32dfb67a512 WatchSource:0}: Error finding container 74130723b4eebe0c2c25bd5a181e11a0f3bcffec218791490249c32dfb67a512: Status 404 returned error can't find the container with id 74130723b4eebe0c2c25bd5a181e11a0f3bcffec218791490249c32dfb67a512
Jul 12 17:03:13.331447 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.331429    2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 12 17:03:13.342619 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.342598    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xnbt\" (UniqueName: \"kubernetes.io/projected/ecab5f05-562e-47bc-9c5a-04893ab8a903-kube-api-access-6xnbt\") pod \"multus-xsskc\" (UID: \"ecab5f05-562e-47bc-9c5a-04893ab8a903\") " pod="openshift-multus/multus-xsskc"
Jul 12 17:03:13.342709 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.342624    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6b618491-4e81-4708-ab1b-2781d1df7c87-node-log\") pod \"ovnkube-node-9xbrj\" (UID: \"6b618491-4e81-4708-ab1b-2781d1df7c87\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xbrj"
Jul 12 17:03:13.342709 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.342639    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6b618491-4e81-4708-ab1b-2781d1df7c87-ovnkube-script-lib\") pod \"ovnkube-node-9xbrj\" (UID: \"6b618491-4e81-4708-ab1b-2781d1df7c87\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xbrj"
Jul 12 17:03:13.342709 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.342653    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8bcfc1cb-b938-4719-a485-dc3acd069a11-etc-modprobe-d\") pod \"tuned-nvmct\" (UID: \"8bcfc1cb-b938-4719-a485-dc3acd069a11\") " pod="openshift-cluster-node-tuning-operator/tuned-nvmct"
Jul 12 17:03:13.342709 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.342674    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/59055e87-8c03-4225-97e1-62114754af67-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ltcld\" (UID: \"59055e87-8c03-4225-97e1-62114754af67\") " pod="openshift-multus/multus-additional-cni-plugins-ltcld"
Jul 12 17:03:13.342909 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.342706    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6b618491-4e81-4708-ab1b-2781d1df7c87-node-log\") pod \"ovnkube-node-9xbrj\" (UID: \"6b618491-4e81-4708-ab1b-2781d1df7c87\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xbrj"
Jul 12 17:03:13.342909 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.342724    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fcdb5e54-4227-4aed-bcb8-b019fc5dfd86-hosts-file\") pod \"node-resolver-zlfcq\" (UID: \"fcdb5e54-4227-4aed-bcb8-b019fc5dfd86\") " pod="openshift-dns/node-resolver-zlfcq"
Jul 12 17:03:13.342909 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.342752    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8dc74bb2-e60b-49c3-9d2a-0bf12cdbc40a-etc-selinux\") pod \"aws-ebs-csi-driver-node-smtm5\" (UID: \"8dc74bb2-e60b-49c3-9d2a-0bf12cdbc40a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-smtm5"
Jul 12 17:03:13.342909 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.342781    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8bcfc1cb-b938-4719-a485-dc3acd069a11-etc-modprobe-d\") pod \"tuned-nvmct\" (UID: \"8bcfc1cb-b938-4719-a485-dc3acd069a11\") " pod="openshift-cluster-node-tuning-operator/tuned-nvmct"
Jul 12 17:03:13.342909 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.342796    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8dc74bb2-e60b-49c3-9d2a-0bf12cdbc40a-etc-selinux\") pod \"aws-ebs-csi-driver-node-smtm5\" (UID: \"8dc74bb2-e60b-49c3-9d2a-0bf12cdbc40a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-smtm5"
Jul 12 17:03:13.342909 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.342822    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fcdb5e54-4227-4aed-bcb8-b019fc5dfd86-hosts-file\") pod \"node-resolver-zlfcq\" (UID: \"fcdb5e54-4227-4aed-bcb8-b019fc5dfd86\") " pod="openshift-dns/node-resolver-zlfcq"
Jul 12 17:03:13.342909 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.342862    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6sf9\" (UniqueName: \"kubernetes.io/projected/259ec8ed-10bd-4fae-8f7f-97409bd4f54b-kube-api-access-n6sf9\") pod \"network-check-target-gxtt8\" (UID: \"259ec8ed-10bd-4fae-8f7f-97409bd4f54b\") " pod="openshift-network-diagnostics/network-check-target-gxtt8"
Jul 12 17:03:13.342909 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.342897    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ecab5f05-562e-47bc-9c5a-04893ab8a903-host-var-lib-cni-bin\") pod \"multus-xsskc\" (UID: \"ecab5f05-562e-47bc-9c5a-04893ab8a903\") " pod="openshift-multus/multus-xsskc"
Jul 12 17:03:13.343257 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.342919    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6b618491-4e81-4708-ab1b-2781d1df7c87-host-kubelet\") pod \"ovnkube-node-9xbrj\" (UID: \"6b618491-4e81-4708-ab1b-2781d1df7c87\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xbrj"
Jul 12 17:03:13.343257 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.342943    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6b618491-4e81-4708-ab1b-2781d1df7c87-host-slash\") pod \"ovnkube-node-9xbrj\" (UID: \"6b618491-4e81-4708-ab1b-2781d1df7c87\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xbrj"
Jul 12 17:03:13.343257 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.342966    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8bcfc1cb-b938-4719-a485-dc3acd069a11-host\") pod \"tuned-nvmct\" (UID: \"8bcfc1cb-b938-4719-a485-dc3acd069a11\") " pod="openshift-cluster-node-tuning-operator/tuned-nvmct"
Jul 12 17:03:13.343257 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.342991    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/59055e87-8c03-4225-97e1-62114754af67-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ltcld\" (UID: \"59055e87-8c03-4225-97e1-62114754af67\") " pod="openshift-multus/multus-additional-cni-plugins-ltcld"
Jul 12 17:03:13.343257 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.342993    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6b618491-4e81-4708-ab1b-2781d1df7c87-host-kubelet\") pod \"ovnkube-node-9xbrj\" (UID: \"6b618491-4e81-4708-ab1b-2781d1df7c87\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xbrj"
Jul 12 17:03:13.343257 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.342997    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6b618491-4e81-4708-ab1b-2781d1df7c87-host-slash\") pod \"ovnkube-node-9xbrj\" (UID: \"6b618491-4e81-4708-ab1b-2781d1df7c87\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xbrj"
Jul 12 17:03:13.343257 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.342987    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ecab5f05-562e-47bc-9c5a-04893ab8a903-host-var-lib-cni-bin\") pod \"multus-xsskc\" (UID: \"ecab5f05-562e-47bc-9c5a-04893ab8a903\") " pod="openshift-multus/multus-xsskc"
Jul 12 17:03:13.343257 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.343017    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c8fd63f2-13ee-4231-8ea3-a83dd90f365a-iptables-alerter-script\") pod \"iptables-alerter-9bc6r\" (UID: \"c8fd63f2-13ee-4231-8ea3-a83dd90f365a\") " pod="openshift-network-operator/iptables-alerter-9bc6r"
Jul 12 17:03:13.343257 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.343029    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8bcfc1cb-b938-4719-a485-dc3acd069a11-host\") pod \"tuned-nvmct\" (UID: \"8bcfc1cb-b938-4719-a485-dc3acd069a11\") " pod="openshift-cluster-node-tuning-operator/tuned-nvmct"
Jul 12 17:03:13.343257 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.343041    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8dc74bb2-e60b-49c3-9d2a-0bf12cdbc40a-registration-dir\") pod \"aws-ebs-csi-driver-node-smtm5\" (UID: \"8dc74bb2-e60b-49c3-9d2a-0bf12cdbc40a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-smtm5"
Jul 12 17:03:13.343257 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.343101    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8dc74bb2-e60b-49c3-9d2a-0bf12cdbc40a-registration-dir\") pod \"aws-ebs-csi-driver-node-smtm5\" (UID: \"8dc74bb2-e60b-49c3-9d2a-0bf12cdbc40a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-smtm5"
Jul 12 17:03:13.343257 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.343118    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ecab5f05-562e-47bc-9c5a-04893ab8a903-cni-binary-copy\") pod \"multus-xsskc\" (UID: \"ecab5f05-562e-47bc-9c5a-04893ab8a903\") " pod="openshift-multus/multus-xsskc"
Jul 12 17:03:13.343257 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.343137    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ecab5f05-562e-47bc-9c5a-04893ab8a903-multus-socket-dir-parent\") pod \"multus-xsskc\" (UID: \"ecab5f05-562e-47bc-9c5a-04893ab8a903\") " pod="openshift-multus/multus-xsskc"
Jul 12 17:03:13.343257 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.343135    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/59055e87-8c03-4225-97e1-62114754af67-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ltcld\" (UID: \"59055e87-8c03-4225-97e1-62114754af67\") " pod="openshift-multus/multus-additional-cni-plugins-ltcld"
Jul 12 17:03:13.343257 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.343152    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ecab5f05-562e-47bc-9c5a-04893ab8a903-host-run-k8s-cni-cncf-io\") pod \"multus-xsskc\" (UID: \"ecab5f05-562e-47bc-9c5a-04893ab8a903\") " pod="openshift-multus/multus-xsskc"
Jul 12 17:03:13.343257 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.343177    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b618491-4e81-4708-ab1b-2781d1df7c87-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9xbrj\" (UID: \"6b618491-4e81-4708-ab1b-2781d1df7c87\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xbrj"
Jul 12 17:03:13.343257 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.343220    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ecab5f05-562e-47bc-9c5a-04893ab8a903-host-run-k8s-cni-cncf-io\") pod \"multus-xsskc\" (UID: \"ecab5f05-562e-47bc-9c5a-04893ab8a903\") " pod="openshift-multus/multus-xsskc"
Jul 12 17:03:13.344079 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.343221    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b618491-4e81-4708-ab1b-2781d1df7c87-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9xbrj\" (UID: \"6b618491-4e81-4708-ab1b-2781d1df7c87\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xbrj"
Jul 12 17:03:13.344079 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.343239    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8bcfc1cb-b938-4719-a485-dc3acd069a11-lib-modules\") pod \"tuned-nvmct\" (UID: \"8bcfc1cb-b938-4719-a485-dc3acd069a11\") " pod="openshift-cluster-node-tuning-operator/tuned-nvmct"
Jul 12 17:03:13.344079 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.343204    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ecab5f05-562e-47bc-9c5a-04893ab8a903-multus-socket-dir-parent\") pod \"multus-xsskc\" (UID: \"ecab5f05-562e-47bc-9c5a-04893ab8a903\") " pod="openshift-multus/multus-xsskc"
Jul 12 17:03:13.344079 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.343265    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ecab5f05-562e-47bc-9c5a-04893ab8a903-os-release\") pod \"multus-xsskc\" (UID: \"ecab5f05-562e-47bc-9c5a-04893ab8a903\") " pod="openshift-multus/multus-xsskc"
Jul 12 17:03:13.344079 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.343263    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/59055e87-8c03-4225-97e1-62114754af67-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ltcld\" (UID: \"59055e87-8c03-4225-97e1-62114754af67\") " pod="openshift-multus/multus-additional-cni-plugins-ltcld"
Jul 12 17:03:13.344079 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.343295    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ecab5f05-562e-47bc-9c5a-04893ab8a903-host-run-netns\") pod \"multus-xsskc\" (UID: \"ecab5f05-562e-47bc-9c5a-04893ab8a903\") " pod="openshift-multus/multus-xsskc"
Jul 12 17:03:13.344079 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.343316    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6b618491-4e81-4708-ab1b-2781d1df7c87-ovnkube-script-lib\") pod \"ovnkube-node-9xbrj\" (UID: \"6b618491-4e81-4708-ab1b-2781d1df7c87\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xbrj"
Jul 12 17:03:13.344079 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.343320    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ecab5f05-562e-47bc-9c5a-04893ab8a903-multus-conf-dir\") pod \"multus-xsskc\" (UID: \"ecab5f05-562e-47bc-9c5a-04893ab8a903\") " pod="openshift-multus/multus-xsskc"
Jul 12 17:03:13.344079 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.343342    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ecab5f05-562e-47bc-9c5a-04893ab8a903-os-release\") pod \"multus-xsskc\" (UID: \"ecab5f05-562e-47bc-9c5a-04893ab8a903\") " pod="openshift-multus/multus-xsskc"
Jul 12 17:03:13.344079 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.343350    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ecab5f05-562e-47bc-9c5a-04893ab8a903-multus-conf-dir\") pod \"multus-xsskc\" (UID: \"ecab5f05-562e-47bc-9c5a-04893ab8a903\") " pod="openshift-multus/multus-xsskc"
Jul 12 17:03:13.344079 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.343361    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6b618491-4e81-4708-ab1b-2781d1df7c87-ovn-node-metrics-cert\") pod \"ovnkube-node-9xbrj\" (UID: \"6b618491-4e81-4708-ab1b-2781d1df7c87\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xbrj"
Jul 12 17:03:13.344079 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.343378    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8bcfc1cb-b938-4719-a485-dc3acd069a11-etc-systemd\") pod \"tuned-nvmct\" (UID: \"8bcfc1cb-b938-4719-a485-dc3acd069a11\") " pod="openshift-cluster-node-tuning-operator/tuned-nvmct"
Jul 12 17:03:13.344079 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.343415    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8bcfc1cb-b938-4719-a485-dc3acd069a11-etc-systemd\") pod \"tuned-nvmct\" (UID: \"8bcfc1cb-b938-4719-a485-dc3acd069a11\") " pod="openshift-cluster-node-tuning-operator/tuned-nvmct"
Jul 12 17:03:13.344079 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.343360    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8bcfc1cb-b938-4719-a485-dc3acd069a11-lib-modules\") pod \"tuned-nvmct\" (UID: \"8bcfc1cb-b938-4719-a485-dc3acd069a11\") " pod="openshift-cluster-node-tuning-operator/tuned-nvmct"
Jul 12 17:03:13.344079 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.343379    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ecab5f05-562e-47bc-9c5a-04893ab8a903-host-run-netns\") pod \"multus-xsskc\" (UID: \"ecab5f05-562e-47bc-9c5a-04893ab8a903\") " pod="openshift-multus/multus-xsskc"
Jul 12 17:03:13.344079 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.343416    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/577e3a93-0560-4099-bddb-dd55563d002d-original-pull-secret\") pod \"global-pull-secret-syncer-mkw2b\" (UID: \"577e3a93-0560-4099-bddb-dd55563d002d\") " pod="kube-system/global-pull-secret-syncer-mkw2b"
Jul 12 17:03:13.344079 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.343476    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9vfd\" (UniqueName: \"kubernetes.io/projected/098853ee-bad5-44d4-b2ac-bc4aa97b5d1c-kube-api-access-t9vfd\") pod \"node-ca-gpzg5\" (UID: \"098853ee-bad5-44d4-b2ac-bc4aa97b5d1c\") " pod="openshift-image-registry/node-ca-gpzg5"
Jul 12 17:03:13.344079 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:03:13.343482    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 12 17:03:13.344953 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.343524    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8bcfc1cb-b938-4719-a485-dc3acd069a11-run\") pod \"tuned-nvmct\" (UID: \"8bcfc1cb-b938-4719-a485-dc3acd069a11\") " pod="openshift-cluster-node-tuning-operator/tuned-nvmct"
Jul 12 17:03:13.344953 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:03:13.343545    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/577e3a93-0560-4099-bddb-dd55563d002d-original-pull-secret podName:577e3a93-0560-4099-bddb-dd55563d002d nodeName:}" failed. No retries permitted until 2026-07-12 17:03:13.843526319 +0000 UTC m=+1.963843104 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/577e3a93-0560-4099-bddb-dd55563d002d-original-pull-secret") pod "global-pull-secret-syncer-mkw2b" (UID: "577e3a93-0560-4099-bddb-dd55563d002d") : object "kube-system"/"original-pull-secret" not registered
Jul 12 17:03:13.344953 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.343574    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/59055e87-8c03-4225-97e1-62114754af67-cnibin\") pod \"multus-additional-cni-plugins-ltcld\" (UID: \"59055e87-8c03-4225-97e1-62114754af67\") " pod="openshift-multus/multus-additional-cni-plugins-ltcld"
Jul 12 17:03:13.344953 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.343568    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8bcfc1cb-b938-4719-a485-dc3acd069a11-run\") pod \"tuned-nvmct\" (UID: \"8bcfc1cb-b938-4719-a485-dc3acd069a11\") " pod="openshift-cluster-node-tuning-operator/tuned-nvmct"
Jul 12 17:03:13.344953 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.343597    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c8fd63f2-13ee-4231-8ea3-a83dd90f365a-iptables-alerter-script\") pod \"iptables-alerter-9bc6r\" (UID: \"c8fd63f2-13ee-4231-8ea3-a83dd90f365a\") " pod="openshift-network-operator/iptables-alerter-9bc6r"
Jul 12 17:03:13.344953 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.343602    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8htkx\" (UniqueName: \"kubernetes.io/projected/59055e87-8c03-4225-97e1-62114754af67-kube-api-access-8htkx\") pod \"multus-additional-cni-plugins-ltcld\" (UID: \"59055e87-8c03-4225-97e1-62114754af67\") " pod="openshift-multus/multus-additional-cni-plugins-ltcld"
Jul 12 17:03:13.344953 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.343645    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/59055e87-8c03-4225-97e1-62114754af67-cnibin\") pod \"multus-additional-cni-plugins-ltcld\" (UID: \"59055e87-8c03-4225-97e1-62114754af67\") " pod="openshift-multus/multus-additional-cni-plugins-ltcld"
Jul 12 17:03:13.344953 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.343655    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6b618491-4e81-4708-ab1b-2781d1df7c87-run-ovn\") pod \"ovnkube-node-9xbrj\" (UID: \"6b618491-4e81-4708-ab1b-2781d1df7c87\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xbrj"
Jul 12 17:03:13.344953 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.343660    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ecab5f05-562e-47bc-9c5a-04893ab8a903-cni-binary-copy\") pod \"multus-xsskc\" (UID: \"ecab5f05-562e-47bc-9c5a-04893ab8a903\") " pod="openshift-multus/multus-xsskc"
Jul 12 17:03:13.344953 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.343703    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b618491-4e81-4708-ab1b-2781d1df7c87-host-run-ovn-kubernetes\") pod \"ovnkube-node-9xbrj\" (UID: \"6b618491-4e81-4708-ab1b-2781d1df7c87\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xbrj"
Jul 12 17:03:13.344953 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.343733    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8bcfc1cb-b938-4719-a485-dc3acd069a11-etc-sysctl-conf\") pod \"tuned-nvmct\" (UID: \"8bcfc1cb-b938-4719-a485-dc3acd069a11\") " pod="openshift-cluster-node-tuning-operator/tuned-nvmct"
Jul 12 17:03:13.344953 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.343705    2569 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 12 17:03:13.344953 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.343755    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b618491-4e81-4708-ab1b-2781d1df7c87-host-run-ovn-kubernetes\") pod \"ovnkube-node-9xbrj\" (UID: \"6b618491-4e81-4708-ab1b-2781d1df7c87\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xbrj"
Jul 12 17:03:13.344953 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.343766    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8bcfc1cb-b938-4719-a485-dc3acd069a11-etc-tuned\") pod \"tuned-nvmct\" (UID: \"8bcfc1cb-b938-4719-a485-dc3acd069a11\") " pod="openshift-cluster-node-tuning-operator/tuned-nvmct"
Jul 12 17:03:13.344953 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.343785    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6b618491-4e81-4708-ab1b-2781d1df7c87-run-ovn\") pod \"ovnkube-node-9xbrj\" (UID: \"6b618491-4e81-4708-ab1b-2781d1df7c87\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xbrj"
Jul 12 17:03:13.344953 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.343789    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/59055e87-8c03-4225-97e1-62114754af67-os-release\") pod \"multus-additional-cni-plugins-ltcld\" (UID: \"59055e87-8c03-4225-97e1-62114754af67\") " pod="openshift-multus/multus-additional-cni-plugins-ltcld"
Jul 12 17:03:13.344953 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.343818    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/59055e87-8c03-4225-97e1-62114754af67-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ltcld\" (UID: \"59055e87-8c03-4225-97e1-62114754af67\") " pod="openshift-multus/multus-additional-cni-plugins-ltcld"
Jul 12 17:03:13.345617 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.343841    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hk6x9\" (UniqueName: \"kubernetes.io/projected/fcdb5e54-4227-4aed-bcb8-b019fc5dfd86-kube-api-access-hk6x9\") pod \"node-resolver-zlfcq\" (UID: \"fcdb5e54-4227-4aed-bcb8-b019fc5dfd86\") " pod="openshift-dns/node-resolver-zlfcq"
Jul 12 17:03:13.345617 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.343869    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6b618491-4e81-4708-ab1b-2781d1df7c87-host-run-netns\") pod \"ovnkube-node-9xbrj\" (UID: \"6b618491-4e81-4708-ab1b-2781d1df7c87\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xbrj"
Jul 12 17:03:13.345617 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.343878    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8bcfc1cb-b938-4719-a485-dc3acd069a11-etc-sysctl-conf\") pod \"tuned-nvmct\" (UID: \"8bcfc1cb-b938-4719-a485-dc3acd069a11\") " pod="openshift-cluster-node-tuning-operator/tuned-nvmct"
Jul 12 17:03:13.345617 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.343891    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2grnc\" (UniqueName: \"kubernetes.io/projected/6b618491-4e81-4708-ab1b-2781d1df7c87-kube-api-access-2grnc\") pod \"ovnkube-node-9xbrj\" (UID: \"6b618491-4e81-4708-ab1b-2781d1df7c87\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xbrj"
Jul 12 17:03:13.345617 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.343903    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/59055e87-8c03-4225-97e1-62114754af67-os-release\") pod \"multus-additional-cni-plugins-ltcld\" (UID: \"59055e87-8c03-4225-97e1-62114754af67\") " pod="openshift-multus/multus-additional-cni-plugins-ltcld"
Jul 12 17:03:13.345617 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.343914    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/59055e87-8c03-4225-97e1-62114754af67-cni-binary-copy\") pod \"multus-additional-cni-plugins-ltcld\" (UID: \"59055e87-8c03-4225-97e1-62114754af67\") " pod="openshift-multus/multus-additional-cni-plugins-ltcld"
Jul 12 17:03:13.345617 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.343930    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6b618491-4e81-4708-ab1b-2781d1df7c87-host-run-netns\") pod \"ovnkube-node-9xbrj\" (UID: \"6b618491-4e81-4708-ab1b-2781d1df7c87\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xbrj"
Jul 12 17:03:13.345617 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.343938    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8dc74bb2-e60b-49c3-9d2a-0bf12cdbc40a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-smtm5\" (UID: \"8dc74bb2-e60b-49c3-9d2a-0bf12cdbc40a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-smtm5"
Jul 12 17:03:13.345617 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.343973    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8dc74bb2-e60b-49c3-9d2a-0bf12cdbc40a-socket-dir\") pod \"aws-ebs-csi-driver-node-smtm5\" (UID: \"8dc74bb2-e60b-49c3-9d2a-0bf12cdbc40a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-smtm5"
Jul 12 17:03:13.345617 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.343997    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsprf\" (UniqueName: \"kubernetes.io/projected/8dc74bb2-e60b-49c3-9d2a-0bf12cdbc40a-kube-api-access-lsprf\") pod \"aws-ebs-csi-driver-node-smtm5\" (UID: \"8dc74bb2-e60b-49c3-9d2a-0bf12cdbc40a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-smtm5"
Jul 12 17:03:13.345617 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.344025    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ecab5f05-562e-47bc-9c5a-04893ab8a903-system-cni-dir\") pod \"multus-xsskc\" (UID: \"ecab5f05-562e-47bc-9c5a-04893ab8a903\") " pod="openshift-multus/multus-xsskc"
Jul 12 17:03:13.345617 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.344057    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ecab5f05-562e-47bc-9c5a-04893ab8a903-host-var-lib-cni-multus\") pod \"multus-xsskc\" (UID: \"ecab5f05-562e-47bc-9c5a-04893ab8a903\") " pod="openshift-multus/multus-xsskc"
Jul 12 17:03:13.345617 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.344079    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6b618491-4e81-4708-ab1b-2781d1df7c87-etc-openvswitch\") pod \"ovnkube-node-9xbrj\" (UID: \"6b618491-4e81-4708-ab1b-2781d1df7c87\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xbrj"
Jul 12 17:03:13.345617 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.344111    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6b618491-4e81-4708-ab1b-2781d1df7c87-log-socket\") pod \"ovnkube-node-9xbrj\" (UID: \"6b618491-4e81-4708-ab1b-2781d1df7c87\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xbrj"
Jul 12 17:03:13.345617 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.344140    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8bcfc1cb-b938-4719-a485-dc3acd069a11-etc-sysconfig\") pod \"tuned-nvmct\" (UID: \"8bcfc1cb-b938-4719-a485-dc3acd069a11\") " pod="openshift-cluster-node-tuning-operator/tuned-nvmct"
Jul 12 17:03:13.345617 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.344167    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8dc74bb2-e60b-49c3-9d2a-0bf12cdbc40a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-smtm5\" (UID: \"8dc74bb2-e60b-49c3-9d2a-0bf12cdbc40a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-smtm5"
Jul 12 17:03:13.345617 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.344176    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbh7x\" (UniqueName: \"kubernetes.io/projected/c8fd63f2-13ee-4231-8ea3-a83dd90f365a-kube-api-access-jbh7x\") pod \"iptables-alerter-9bc6r\" (UID: \"c8fd63f2-13ee-4231-8ea3-a83dd90f365a\") " pod="openshift-network-operator/iptables-alerter-9bc6r"
Jul 12 17:03:13.346328 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.344200    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/577e3a93-0560-4099-bddb-dd55563d002d-kubelet-config\") pod \"global-pull-secret-syncer-mkw2b\" (UID: \"577e3a93-0560-4099-bddb-dd55563d002d\") " pod="kube-system/global-pull-secret-syncer-mkw2b"
Jul 12 17:03:13.346328 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.344222    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fcdb5e54-4227-4aed-bcb8-b019fc5dfd86-tmp-dir\") pod \"node-resolver-zlfcq\" (UID: \"fcdb5e54-4227-4aed-bcb8-b019fc5dfd86\") " pod="openshift-dns/node-resolver-zlfcq"
Jul 12 17:03:13.346328 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.344244    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ecab5f05-562e-47bc-9c5a-04893ab8a903-cnibin\") pod \"multus-xsskc\" (UID: \"ecab5f05-562e-47bc-9c5a-04893ab8a903\") " pod="openshift-multus/multus-xsskc"
Jul 12 17:03:13.346328 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.344267    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8bcfc1cb-b938-4719-a485-dc3acd069a11-var-lib-kubelet\") pod \"tuned-nvmct\" (UID: \"8bcfc1cb-b938-4719-a485-dc3acd069a11\") " pod="openshift-cluster-node-tuning-operator/tuned-nvmct"
Jul 12 17:03:13.346328 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.344296    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/098853ee-bad5-44d4-b2ac-bc4aa97b5d1c-host\") pod \"node-ca-gpzg5\" (UID: \"098853ee-bad5-44d4-b2ac-bc4aa97b5d1c\") " pod="openshift-image-registry/node-ca-gpzg5"
Jul 12 17:03:13.346328 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.344319    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/098853ee-bad5-44d4-b2ac-bc4aa97b5d1c-serviceca\") pod \"node-ca-gpzg5\" (UID: \"098853ee-bad5-44d4-b2ac-bc4aa97b5d1c\") " pod="openshift-image-registry/node-ca-gpzg5"
Jul 12 17:03:13.346328 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.344343    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6b618491-4e81-4708-ab1b-2781d1df7c87-systemd-units\") pod \"ovnkube-node-9xbrj\" (UID: \"6b618491-4e81-4708-ab1b-2781d1df7c87\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xbrj"
Jul 12 17:03:13.346328 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.344367    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6b618491-4e81-4708-ab1b-2781d1df7c87-var-lib-openvswitch\") pod \"ovnkube-node-9xbrj\" (UID: \"6b618491-4e81-4708-ab1b-2781d1df7c87\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xbrj"
Jul 12 17:03:13.346328 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.344378    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/59055e87-8c03-4225-97e1-62114754af67-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ltcld\" (UID: \"59055e87-8c03-4225-97e1-62114754af67\") " pod="openshift-multus/multus-additional-cni-plugins-ltcld"
Jul 12 17:03:13.346328 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.344397    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8dc74bb2-e60b-49c3-9d2a-0bf12cdbc40a-sys-fs\") pod \"aws-ebs-csi-driver-node-smtm5\" (UID: \"8dc74bb2-e60b-49c3-9d2a-0bf12cdbc40a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-smtm5"
Jul 12 17:03:13.346328 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.344432    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9gdk\" (UniqueName: \"kubernetes.io/projected/f1d6c71b-d9fb-4f2a-9a81-5e8f0f8a6c13-kube-api-access-x9gdk\") pod \"network-metrics-daemon-h442x\" (UID: \"f1d6c71b-d9fb-4f2a-9a81-5e8f0f8a6c13\") " pod="openshift-multus/network-metrics-daemon-h442x"
Jul 12 17:03:13.346328 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.344434    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/59055e87-8c03-4225-97e1-62114754af67-cni-binary-copy\") pod \"multus-additional-cni-plugins-ltcld\" (UID: \"59055e87-8c03-4225-97e1-62114754af67\") " pod="openshift-multus/multus-additional-cni-plugins-ltcld"
Jul 12 17:03:13.346328 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.344453    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ecab5f05-562e-47bc-9c5a-04893ab8a903-system-cni-dir\") pod \"multus-xsskc\" (UID: \"ecab5f05-562e-47bc-9c5a-04893ab8a903\") " pod="openshift-multus/multus-xsskc"
Jul 12 17:03:13.346328 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.344458    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ecab5f05-562e-47bc-9c5a-04893ab8a903-multus-cni-dir\") pod \"multus-xsskc\" (UID: \"ecab5f05-562e-47bc-9c5a-04893ab8a903\") " pod="openshift-multus/multus-xsskc"
Jul 12 17:03:13.346328 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.344272    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8dc74bb2-e60b-49c3-9d2a-0bf12cdbc40a-socket-dir\") pod \"aws-ebs-csi-driver-node-smtm5\" (UID: \"8dc74bb2-e60b-49c3-9d2a-0bf12cdbc40a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-smtm5"
Jul 12 17:03:13.346328 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.344483    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ecab5f05-562e-47bc-9c5a-04893ab8a903-multus-daemon-config\") pod \"multus-xsskc\" (UID: \"ecab5f05-562e-47bc-9c5a-04893ab8a903\") " pod="openshift-multus/multus-xsskc"
Jul 12 17:03:13.346328 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.344499    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ecab5f05-562e-47bc-9c5a-04893ab8a903-host-var-lib-cni-multus\") pod \"multus-xsskc\" (UID: \"ecab5f05-562e-47bc-9c5a-04893ab8a903\") " pod="openshift-multus/multus-xsskc"
Jul 12 17:03:13.346964 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.344525    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6b618491-4e81-4708-ab1b-2781d1df7c87-log-socket\") pod \"ovnkube-node-9xbrj\" (UID: \"6b618491-4e81-4708-ab1b-2781d1df7c87\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xbrj"
Jul 12 17:03:13.346964 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.344530    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6b618491-4e81-4708-ab1b-2781d1df7c87-run-systemd\") pod \"ovnkube-node-9xbrj\" (UID: \"6b618491-4e81-4708-ab1b-2781d1df7c87\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xbrj"
Jul 12 17:03:13.346964 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.344544    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8bcfc1cb-b938-4719-a485-dc3acd069a11-etc-sysconfig\") pod \"tuned-nvmct\" (UID: \"8bcfc1cb-b938-4719-a485-dc3acd069a11\") " pod="openshift-cluster-node-tuning-operator/tuned-nvmct"
Jul 12 17:03:13.346964 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.344562    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6b618491-4e81-4708-ab1b-2781d1df7c87-var-lib-openvswitch\") pod \"ovnkube-node-9xbrj\" (UID: \"6b618491-4e81-4708-ab1b-2781d1df7c87\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xbrj"
Jul 12 17:03:13.346964 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.344575    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6b618491-4e81-4708-ab1b-2781d1df7c87-run-systemd\") pod \"ovnkube-node-9xbrj\" (UID: \"6b618491-4e81-4708-ab1b-2781d1df7c87\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xbrj"
Jul 12 17:03:13.346964 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.344601    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/098853ee-bad5-44d4-b2ac-bc4aa97b5d1c-host\") pod \"node-ca-gpzg5\" (UID: \"098853ee-bad5-44d4-b2ac-bc4aa97b5d1c\") " pod="openshift-image-registry/node-ca-gpzg5"
Jul 12 17:03:13.346964 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.344604    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6b618491-4e81-4708-ab1b-2781d1df7c87-systemd-units\") pod \"ovnkube-node-9xbrj\" (UID: \"6b618491-4e81-4708-ab1b-2781d1df7c87\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xbrj"
Jul 12 17:03:13.346964 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.344627    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8dc74bb2-e60b-49c3-9d2a-0bf12cdbc40a-sys-fs\") pod \"aws-ebs-csi-driver-node-smtm5\" (UID: \"8dc74bb2-e60b-49c3-9d2a-0bf12cdbc40a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-smtm5"
Jul 12 17:03:13.346964 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.344659    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/577e3a93-0560-4099-bddb-dd55563d002d-kubelet-config\") pod \"global-pull-secret-syncer-mkw2b\" (UID: \"577e3a93-0560-4099-bddb-dd55563d002d\") " pod="kube-system/global-pull-secret-syncer-mkw2b"
Jul 12 17:03:13.346964 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.344849    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ecab5f05-562e-47bc-9c5a-04893ab8a903-multus-cni-dir\") pod \"multus-xsskc\" (UID: \"ecab5f05-562e-47bc-9c5a-04893ab8a903\") " pod="openshift-multus/multus-xsskc"
Jul 12 17:03:13.346964 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.344867    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6b618491-4e81-4708-ab1b-2781d1df7c87-host-cni-netd\") pod \"ovnkube-node-9xbrj\" (UID: \"6b618491-4e81-4708-ab1b-2781d1df7c87\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xbrj"
Jul 12 17:03:13.346964 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.344905    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ecab5f05-562e-47bc-9c5a-04893ab8a903-cnibin\") pod \"multus-xsskc\" (UID: \"ecab5f05-562e-47bc-9c5a-04893ab8a903\") " pod="openshift-multus/multus-xsskc"
Jul 12 17:03:13.346964 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.344905    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/577e3a93-0560-4099-bddb-dd55563d002d-dbus\") pod \"global-pull-secret-syncer-mkw2b\" (UID: \"577e3a93-0560-4099-bddb-dd55563d002d\") " pod="kube-system/global-pull-secret-syncer-mkw2b"
Jul 12 17:03:13.346964 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.344921    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fcdb5e54-4227-4aed-bcb8-b019fc5dfd86-tmp-dir\") pod \"node-resolver-zlfcq\" (UID: \"fcdb5e54-4227-4aed-bcb8-b019fc5dfd86\") " pod="openshift-dns/node-resolver-zlfcq"
Jul 12 17:03:13.346964 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.344945    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6b618491-4e81-4708-ab1b-2781d1df7c87-host-cni-netd\") pod \"ovnkube-node-9xbrj\" (UID: \"6b618491-4e81-4708-ab1b-2781d1df7c87\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xbrj"
Jul 12 17:03:13.346964 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.344940    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6b618491-4e81-4708-ab1b-2781d1df7c87-env-overrides\") pod \"ovnkube-node-9xbrj\" (UID: \"6b618491-4e81-4708-ab1b-2781d1df7c87\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xbrj"
Jul 12 17:03:13.346964 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.344458    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6b618491-4e81-4708-ab1b-2781d1df7c87-etc-openvswitch\") pod \"ovnkube-node-9xbrj\" (UID: \"6b618491-4e81-4708-ab1b-2781d1df7c87\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xbrj"
Jul 12 17:03:13.346964 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.344988    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8bcfc1cb-b938-4719-a485-dc3acd069a11-etc-kubernetes\") pod \"tuned-nvmct\" (UID: \"8bcfc1cb-b938-4719-a485-dc3acd069a11\") " pod="openshift-cluster-node-tuning-operator/tuned-nvmct"
Jul 12 17:03:13.347518 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.344993    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8bcfc1cb-b938-4719-a485-dc3acd069a11-var-lib-kubelet\") pod \"tuned-nvmct\" (UID: \"8bcfc1cb-b938-4719-a485-dc3acd069a11\") " pod="openshift-cluster-node-tuning-operator/tuned-nvmct"
Jul 12 17:03:13.347518 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.345015    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8bcfc1cb-b938-4719-a485-dc3acd069a11-etc-sysctl-d\") pod \"tuned-nvmct\" (UID: \"8bcfc1cb-b938-4719-a485-dc3acd069a11\") " pod="openshift-cluster-node-tuning-operator/tuned-nvmct"
Jul 12 17:03:13.347518 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.345038    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mg5l\" (UniqueName: \"kubernetes.io/projected/8bcfc1cb-b938-4719-a485-dc3acd069a11-kube-api-access-2mg5l\") pod \"tuned-nvmct\" (UID: \"8bcfc1cb-b938-4719-a485-dc3acd069a11\") " pod="openshift-cluster-node-tuning-operator/tuned-nvmct"
Jul 12 17:03:13.347518 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.345067    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5598dd9a-714f-4e7d-9c59-a6e2f61c5a74-konnectivity-ca\") pod \"konnectivity-agent-kjdjh\" (UID: \"5598dd9a-714f-4e7d-9c59-a6e2f61c5a74\") " pod="kube-system/konnectivity-agent-kjdjh"
Jul 12 17:03:13.347518 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.345093    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/577e3a93-0560-4099-bddb-dd55563d002d-dbus\") pod \"global-pull-secret-syncer-mkw2b\" (UID: \"577e3a93-0560-4099-bddb-dd55563d002d\") " pod="kube-system/global-pull-secret-syncer-mkw2b"
Jul 12 17:03:13.347518 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.345109    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6b618491-4e81-4708-ab1b-2781d1df7c87-host-cni-bin\") pod \"ovnkube-node-9xbrj\" (UID: \"6b618491-4e81-4708-ab1b-2781d1df7c87\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xbrj"
Jul 12 17:03:13.347518 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.345136    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ecab5f05-562e-47bc-9c5a-04893ab8a903-host-var-lib-kubelet\") pod \"multus-xsskc\" (UID: \"ecab5f05-562e-47bc-9c5a-04893ab8a903\") " pod="openshift-multus/multus-xsskc"
Jul 12 17:03:13.347518 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.345157    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6b618491-4e81-4708-ab1b-2781d1df7c87-ovnkube-config\") pod \"ovnkube-node-9xbrj\" (UID: \"6b618491-4e81-4708-ab1b-2781d1df7c87\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xbrj"
Jul 12 17:03:13.347518 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.345181    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ecab5f05-562e-47bc-9c5a-04893ab8a903-host-run-multus-certs\") pod \"multus-xsskc\" (UID: \"ecab5f05-562e-47bc-9c5a-04893ab8a903\") " pod="openshift-multus/multus-xsskc"
Jul 12 17:03:13.347518 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.345204    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ecab5f05-562e-47bc-9c5a-04893ab8a903-etc-kubernetes\") pod \"multus-xsskc\" (UID: \"ecab5f05-562e-47bc-9c5a-04893ab8a903\") " pod="openshift-multus/multus-xsskc"
Jul 12 17:03:13.347518 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.345223    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8bcfc1cb-b938-4719-a485-dc3acd069a11-etc-kubernetes\") pod \"tuned-nvmct\" (UID: \"8bcfc1cb-b938-4719-a485-dc3acd069a11\") " pod="openshift-cluster-node-tuning-operator/tuned-nvmct"
Jul 12 17:03:13.347518 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.345227    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6b618491-4e81-4708-ab1b-2781d1df7c87-run-openvswitch\") pod \"ovnkube-node-9xbrj\" (UID: \"6b618491-4e81-4708-ab1b-2781d1df7c87\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xbrj"
Jul 12 17:03:13.347518 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.345249    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8bcfc1cb-b938-4719-a485-dc3acd069a11-tmp\") pod \"tuned-nvmct\" (UID: \"8bcfc1cb-b938-4719-a485-dc3acd069a11\") " pod="openshift-cluster-node-tuning-operator/tuned-nvmct"
Jul 12 17:03:13.347518 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.345264    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ecab5f05-562e-47bc-9c5a-04893ab8a903-host-var-lib-kubelet\") pod \"multus-xsskc\" (UID: \"ecab5f05-562e-47bc-9c5a-04893ab8a903\") " pod="openshift-multus/multus-xsskc"
Jul 12 17:03:13.347518 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.345273    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/59055e87-8c03-4225-97e1-62114754af67-system-cni-dir\") pod \"multus-additional-cni-plugins-ltcld\" (UID: \"59055e87-8c03-4225-97e1-62114754af67\") " pod="openshift-multus/multus-additional-cni-plugins-ltcld"
Jul 12 17:03:13.347518 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.345298    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8dc74bb2-e60b-49c3-9d2a-0bf12cdbc40a-device-dir\") pod \"aws-ebs-csi-driver-node-smtm5\" (UID: \"8dc74bb2-e60b-49c3-9d2a-0bf12cdbc40a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-smtm5"
Jul 12 17:03:13.347518 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.345322    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5598dd9a-714f-4e7d-9c59-a6e2f61c5a74-agent-certs\") pod \"konnectivity-agent-kjdjh\" (UID: \"5598dd9a-714f-4e7d-9c59-a6e2f61c5a74\") " pod="kube-system/konnectivity-agent-kjdjh"
Jul 12 17:03:13.347518 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.345345    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ecab5f05-562e-47bc-9c5a-04893ab8a903-hostroot\") pod \"multus-xsskc\" (UID: \"ecab5f05-562e-47bc-9c5a-04893ab8a903\") " pod="openshift-multus/multus-xsskc"
Jul 12 17:03:13.348200 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.345366    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8bcfc1cb-b938-4719-a485-dc3acd069a11-sys\") pod \"tuned-nvmct\" (UID: \"8bcfc1cb-b938-4719-a485-dc3acd069a11\") " pod="openshift-cluster-node-tuning-operator/tuned-nvmct"
Jul 12 17:03:13.348200 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.345390    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c8fd63f2-13ee-4231-8ea3-a83dd90f365a-host-slash\") pod \"iptables-alerter-9bc6r\" (UID: \"c8fd63f2-13ee-4231-8ea3-a83dd90f365a\") " pod="openshift-network-operator/iptables-alerter-9bc6r"
Jul 12 17:03:13.348200 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.345391    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6b618491-4e81-4708-ab1b-2781d1df7c87-env-overrides\") pod \"ovnkube-node-9xbrj\" (UID: \"6b618491-4e81-4708-ab1b-2781d1df7c87\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xbrj"
Jul 12 17:03:13.348200 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.345414    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1d6c71b-d9fb-4f2a-9a81-5e8f0f8a6c13-metrics-certs\") pod \"network-metrics-daemon-h442x\" (UID: \"f1d6c71b-d9fb-4f2a-9a81-5e8f0f8a6c13\") " pod="openshift-multus/network-metrics-daemon-h442x"
Jul 12 17:03:13.348200 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:03:13.345535    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 12 17:03:13.348200 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:03:13.345592    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1d6c71b-d9fb-4f2a-9a81-5e8f0f8a6c13-metrics-certs podName:f1d6c71b-d9fb-4f2a-9a81-5e8f0f8a6c13 nodeName:}" failed. No retries permitted until 2026-07-12 17:03:13.845574768 +0000 UTC m=+1.965891550 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f1d6c71b-d9fb-4f2a-9a81-5e8f0f8a6c13-metrics-certs") pod "network-metrics-daemon-h442x" (UID: "f1d6c71b-d9fb-4f2a-9a81-5e8f0f8a6c13") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 12 17:03:13.348200 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.345600    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/098853ee-bad5-44d4-b2ac-bc4aa97b5d1c-serviceca\") pod \"node-ca-gpzg5\" (UID: \"098853ee-bad5-44d4-b2ac-bc4aa97b5d1c\") " pod="openshift-image-registry/node-ca-gpzg5"
Jul 12 17:03:13.348200 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.345674    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6b618491-4e81-4708-ab1b-2781d1df7c87-host-cni-bin\") pod \"ovnkube-node-9xbrj\" (UID: \"6b618491-4e81-4708-ab1b-2781d1df7c87\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xbrj"
Jul 12 17:03:13.348200 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.345727    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/59055e87-8c03-4225-97e1-62114754af67-system-cni-dir\") pod \"multus-additional-cni-plugins-ltcld\" (UID: \"59055e87-8c03-4225-97e1-62114754af67\") " pod="openshift-multus/multus-additional-cni-plugins-ltcld"
Jul 12 17:03:13.348200 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.345773    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ecab5f05-562e-47bc-9c5a-04893ab8a903-host-run-multus-certs\") pod \"multus-xsskc\" (UID: \"ecab5f05-562e-47bc-9c5a-04893ab8a903\") " pod="openshift-multus/multus-xsskc"
Jul 12 17:03:13.348200 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.345182    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8bcfc1cb-b938-4719-a485-dc3acd069a11-etc-sysctl-d\") pod \"tuned-nvmct\" (UID: \"8bcfc1cb-b938-4719-a485-dc3acd069a11\") " pod="openshift-cluster-node-tuning-operator/tuned-nvmct"
Jul 12 17:03:13.348200 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.345829    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ecab5f05-562e-47bc-9c5a-04893ab8a903-etc-kubernetes\") pod \"multus-xsskc\" (UID: \"ecab5f05-562e-47bc-9c5a-04893ab8a903\") " pod="openshift-multus/multus-xsskc"
Jul 12 17:03:13.348200 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.345870    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6b618491-4e81-4708-ab1b-2781d1df7c87-run-openvswitch\") pod \"ovnkube-node-9xbrj\" (UID: \"6b618491-4e81-4708-ab1b-2781d1df7c87\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xbrj"
Jul 12 17:03:13.348200 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.345920    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5598dd9a-714f-4e7d-9c59-a6e2f61c5a74-konnectivity-ca\") pod \"konnectivity-agent-kjdjh\" (UID: \"5598dd9a-714f-4e7d-9c59-a6e2f61c5a74\") " pod="kube-system/konnectivity-agent-kjdjh"
Jul 12 17:03:13.348200 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.345980    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ecab5f05-562e-47bc-9c5a-04893ab8a903-hostroot\") pod \"multus-xsskc\" (UID: \"ecab5f05-562e-47bc-9c5a-04893ab8a903\") " pod="openshift-multus/multus-xsskc"
Jul 12 17:03:13.348200 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.346009    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8bcfc1cb-b938-4719-a485-dc3acd069a11-sys\") pod \"tuned-nvmct\" (UID: \"8bcfc1cb-b938-4719-a485-dc3acd069a11\") " pod="openshift-cluster-node-tuning-operator/tuned-nvmct"
Jul 12 17:03:13.348200 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.346043    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8dc74bb2-e60b-49c3-9d2a-0bf12cdbc40a-device-dir\") pod \"aws-ebs-csi-driver-node-smtm5\" (UID: \"8dc74bb2-e60b-49c3-9d2a-0bf12cdbc40a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-smtm5"
Jul 12 17:03:13.348200 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.346056    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c8fd63f2-13ee-4231-8ea3-a83dd90f365a-host-slash\") pod \"iptables-alerter-9bc6r\" (UID: \"c8fd63f2-13ee-4231-8ea3-a83dd90f365a\") " pod="openshift-network-operator/iptables-alerter-9bc6r"
Jul 12 17:03:13.348799 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.346419    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ecab5f05-562e-47bc-9c5a-04893ab8a903-multus-daemon-config\") pod \"multus-xsskc\" (UID: \"ecab5f05-562e-47bc-9c5a-04893ab8a903\") " pod="openshift-multus/multus-xsskc"
Jul 12 17:03:13.348799 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.346691    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8bcfc1cb-b938-4719-a485-dc3acd069a11-etc-tuned\") pod \"tuned-nvmct\" (UID: \"8bcfc1cb-b938-4719-a485-dc3acd069a11\") " pod="openshift-cluster-node-tuning-operator/tuned-nvmct"
Jul 12 17:03:13.348799 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.346728    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6b618491-4e81-4708-ab1b-2781d1df7c87-ovnkube-config\") pod \"ovnkube-node-9xbrj\" (UID: \"6b618491-4e81-4708-ab1b-2781d1df7c87\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xbrj"
Jul 12 17:03:13.348799 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.346801    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6b618491-4e81-4708-ab1b-2781d1df7c87-ovn-node-metrics-cert\") pod \"ovnkube-node-9xbrj\" (UID: \"6b618491-4e81-4708-ab1b-2781d1df7c87\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xbrj"
Jul 12 17:03:13.348799 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.348012    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8bcfc1cb-b938-4719-a485-dc3acd069a11-tmp\") pod \"tuned-nvmct\" (UID: \"8bcfc1cb-b938-4719-a485-dc3acd069a11\") " pod="openshift-cluster-node-tuning-operator/tuned-nvmct"
Jul 12 17:03:13.348799 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.348226    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5598dd9a-714f-4e7d-9c59-a6e2f61c5a74-agent-certs\") pod \"konnectivity-agent-kjdjh\" (UID: \"5598dd9a-714f-4e7d-9c59-a6e2f61c5a74\") " pod="kube-system/konnectivity-agent-kjdjh"
Jul 12 17:03:13.350650 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:03:13.350636    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 12 17:03:13.350650 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:03:13.350651    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 12 17:03:13.350758 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:03:13.350660    2569 projected.go:196] Error preparing data for projected volume kube-api-access-n6sf9 for pod openshift-network-diagnostics/network-check-target-gxtt8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 12 17:03:13.350758 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:03:13.350701    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/259ec8ed-10bd-4fae-8f7f-97409bd4f54b-kube-api-access-n6sf9 podName:259ec8ed-10bd-4fae-8f7f-97409bd4f54b nodeName:}" failed. No retries permitted until 2026-07-12 17:03:13.850688662 +0000 UTC m=+1.971005460 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-n6sf9" (UniqueName: "kubernetes.io/projected/259ec8ed-10bd-4fae-8f7f-97409bd4f54b-kube-api-access-n6sf9") pod "network-check-target-gxtt8" (UID: "259ec8ed-10bd-4fae-8f7f-97409bd4f54b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 12 17:03:13.352675 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.352650    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9gdk\" (UniqueName: \"kubernetes.io/projected/f1d6c71b-d9fb-4f2a-9a81-5e8f0f8a6c13-kube-api-access-x9gdk\") pod \"network-metrics-daemon-h442x\" (UID: \"f1d6c71b-d9fb-4f2a-9a81-5e8f0f8a6c13\") " pod="openshift-multus/network-metrics-daemon-h442x"
Jul 12 17:03:13.353381 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.353359    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsprf\" (UniqueName: \"kubernetes.io/projected/8dc74bb2-e60b-49c3-9d2a-0bf12cdbc40a-kube-api-access-lsprf\") pod \"aws-ebs-csi-driver-node-smtm5\" (UID: \"8dc74bb2-e60b-49c3-9d2a-0bf12cdbc40a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-smtm5"
Jul 12 17:03:13.353777 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.353755    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2grnc\" (UniqueName: \"kubernetes.io/projected/6b618491-4e81-4708-ab1b-2781d1df7c87-kube-api-access-2grnc\") pod \"ovnkube-node-9xbrj\" (UID: \"6b618491-4e81-4708-ab1b-2781d1df7c87\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xbrj"
Jul 12 17:03:13.354120 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.354083    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xnbt\" (UniqueName: \"kubernetes.io/projected/ecab5f05-562e-47bc-9c5a-04893ab8a903-kube-api-access-6xnbt\") pod \"multus-xsskc\" (UID: \"ecab5f05-562e-47bc-9c5a-04893ab8a903\") " pod="openshift-multus/multus-xsskc"
Jul 12 17:03:13.354218 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.354203    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9vfd\" (UniqueName: \"kubernetes.io/projected/098853ee-bad5-44d4-b2ac-bc4aa97b5d1c-kube-api-access-t9vfd\") pod \"node-ca-gpzg5\" (UID: \"098853ee-bad5-44d4-b2ac-bc4aa97b5d1c\") " pod="openshift-image-registry/node-ca-gpzg5"
Jul 12 17:03:13.355432 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.355409    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hk6x9\" (UniqueName: \"kubernetes.io/projected/fcdb5e54-4227-4aed-bcb8-b019fc5dfd86-kube-api-access-hk6x9\") pod \"node-resolver-zlfcq\" (UID: \"fcdb5e54-4227-4aed-bcb8-b019fc5dfd86\") " pod="openshift-dns/node-resolver-zlfcq"
Jul 12 17:03:13.355581 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.355503    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8htkx\" (UniqueName: \"kubernetes.io/projected/59055e87-8c03-4225-97e1-62114754af67-kube-api-access-8htkx\") pod \"multus-additional-cni-plugins-ltcld\" (UID: \"59055e87-8c03-4225-97e1-62114754af67\") " pod="openshift-multus/multus-additional-cni-plugins-ltcld"
Jul 12 17:03:13.355965 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.355930    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mg5l\" (UniqueName: \"kubernetes.io/projected/8bcfc1cb-b938-4719-a485-dc3acd069a11-kube-api-access-2mg5l\") pod \"tuned-nvmct\" (UID: \"8bcfc1cb-b938-4719-a485-dc3acd069a11\") " pod="openshift-cluster-node-tuning-operator/tuned-nvmct"
Jul 12 17:03:13.356037 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.356023    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbh7x\" (UniqueName: \"kubernetes.io/projected/c8fd63f2-13ee-4231-8ea3-a83dd90f365a-kube-api-access-jbh7x\") pod \"iptables-alerter-9bc6r\" (UID: \"c8fd63f2-13ee-4231-8ea3-a83dd90f365a\") " pod="openshift-network-operator/iptables-alerter-9bc6r"
Jul 12 17:03:13.375799 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.375757    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-94.ec2.internal" event={"ID":"bf9d9cd767ec6f08a024f6a8e7fbc81d","Type":"ContainerStarted","Data":"74130723b4eebe0c2c25bd5a181e11a0f3bcffec218791490249c32dfb67a512"}
Jul 12 17:03:13.376763 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.376740    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-94.ec2.internal" event={"ID":"64598ecc4535f6658a01b7cab4f3caeb","Type":"ContainerStarted","Data":"97756e8b166fb30026f6b43a0480bc22549a1a1ca1c721740371aea85631393f"}
Jul 12 17:03:13.541599 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.541542    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-kjdjh"
Jul 12 17:03:13.549383 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:13.549364    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5598dd9a_714f_4e7d_9c59_a6e2f61c5a74.slice/crio-30eb2a74f28528aecdcf2b5bd817ddac927661fc48574d5e591519c5aadaec9e WatchSource:0}: Error finding container 30eb2a74f28528aecdcf2b5bd817ddac927661fc48574d5e591519c5aadaec9e: Status 404 returned error can't find the container with id 30eb2a74f28528aecdcf2b5bd817ddac927661fc48574d5e591519c5aadaec9e
Jul 12 17:03:13.560429 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.560414    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-xsskc"
Jul 12 17:03:13.564096 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.564081    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-9bc6r"
Jul 12 17:03:13.566966 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:13.566936    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecab5f05_562e_47bc_9c5a_04893ab8a903.slice/crio-59949c578e09968c95295ad84bdf9f6786db993d3427b0f884bdbf0d3ad6ff6a WatchSource:0}: Error finding container 59949c578e09968c95295ad84bdf9f6786db993d3427b0f884bdbf0d3ad6ff6a: Status 404 returned error can't find the container with id 59949c578e09968c95295ad84bdf9f6786db993d3427b0f884bdbf0d3ad6ff6a
Jul 12 17:03:13.572644 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:13.572441    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8fd63f2_13ee_4231_8ea3_a83dd90f365a.slice/crio-83e5abb67c537b44e43d064f7e2da0f097331f6ee66a09c893c1d7802bedea8b WatchSource:0}: Error finding container 83e5abb67c537b44e43d064f7e2da0f097331f6ee66a09c893c1d7802bedea8b: Status 404 returned error can't find the container with id 83e5abb67c537b44e43d064f7e2da0f097331f6ee66a09c893c1d7802bedea8b
Jul 12 17:03:13.573351 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.573335    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-9xbrj"
Jul 12 17:03:13.580207 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:13.580189    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b618491_4e81_4708_ab1b_2781d1df7c87.slice/crio-ecce4def78cf504b75a53b58afb94dc17b6cb4edea8c651b5367681ce1bf4bc5 WatchSource:0}: Error finding container ecce4def78cf504b75a53b58afb94dc17b6cb4edea8c651b5367681ce1bf4bc5: Status 404 returned error can't find the container with id ecce4def78cf504b75a53b58afb94dc17b6cb4edea8c651b5367681ce1bf4bc5
Jul 12 17:03:13.592148 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.592129    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-nvmct"
Jul 12 17:03:13.598691 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:13.598673    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bcfc1cb_b938_4719_a485_dc3acd069a11.slice/crio-87fbf994de27e044c9671b311fe672939212decc4d0a3a5c8a2d955ddef4abab WatchSource:0}: Error finding container 87fbf994de27e044c9671b311fe672939212decc4d0a3a5c8a2d955ddef4abab: Status 404 returned error can't find the container with id 87fbf994de27e044c9671b311fe672939212decc4d0a3a5c8a2d955ddef4abab
Jul 12 17:03:13.610026 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.610007    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-zlfcq"
Jul 12 17:03:13.616112 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:13.616095    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcdb5e54_4227_4aed_bcb8_b019fc5dfd86.slice/crio-bff536fc8dfc910c3e1f704aeb2ec9d171f1bbe6b0fb04beccf611e165512d22 WatchSource:0}: Error finding container bff536fc8dfc910c3e1f704aeb2ec9d171f1bbe6b0fb04beccf611e165512d22: Status 404 returned error can't find the container with id bff536fc8dfc910c3e1f704aeb2ec9d171f1bbe6b0fb04beccf611e165512d22
Jul 12 17:03:13.632906 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.632890    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-gpzg5"
Jul 12 17:03:13.638623 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.638603    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-smtm5"
Jul 12 17:03:13.638991 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:13.638971    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod098853ee_bad5_44d4_b2ac_bc4aa97b5d1c.slice/crio-46358fdf14c86d0970b57520cebb2cba4c2c4420d325124b3702d7a65e5a2d79 WatchSource:0}: Error finding container 46358fdf14c86d0970b57520cebb2cba4c2c4420d325124b3702d7a65e5a2d79: Status 404 returned error can't find the container with id 46358fdf14c86d0970b57520cebb2cba4c2c4420d325124b3702d7a65e5a2d79
Jul 12 17:03:13.642882 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.642867    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-ltcld"
Jul 12 17:03:13.645685 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:13.645668    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8dc74bb2_e60b_49c3_9d2a_0bf12cdbc40a.slice/crio-3f2596709c056512fb7558c7b9d8fb591bc65f9c8314593c56d1762d899fb7c3 WatchSource:0}: Error finding container 3f2596709c056512fb7558c7b9d8fb591bc65f9c8314593c56d1762d899fb7c3: Status 404 returned error can't find the container with id 3f2596709c056512fb7558c7b9d8fb591bc65f9c8314593c56d1762d899fb7c3
Jul 12 17:03:13.650044 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:13.650025    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59055e87_8c03_4225_97e1_62114754af67.slice/crio-6c91253dc2d761ef2c80e0b4552c8e082513d4142327989f19da061346eca0fd WatchSource:0}: Error finding container 6c91253dc2d761ef2c80e0b4552c8e082513d4142327989f19da061346eca0fd: Status 404 returned error can't find the container with id 6c91253dc2d761ef2c80e0b4552c8e082513d4142327989f19da061346eca0fd
Jul 12 17:03:13.692310 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.692291    2569 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 12 17:03:13.848091 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.848020    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1d6c71b-d9fb-4f2a-9a81-5e8f0f8a6c13-metrics-certs\") pod \"network-metrics-daemon-h442x\" (UID: \"f1d6c71b-d9fb-4f2a-9a81-5e8f0f8a6c13\") " pod="openshift-multus/network-metrics-daemon-h442x"
Jul 12 17:03:13.848234 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.848091    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/577e3a93-0560-4099-bddb-dd55563d002d-original-pull-secret\") pod \"global-pull-secret-syncer-mkw2b\" (UID: \"577e3a93-0560-4099-bddb-dd55563d002d\") " pod="kube-system/global-pull-secret-syncer-mkw2b"
Jul 12 17:03:13.848234 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:03:13.848215    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 12 17:03:13.848347 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:03:13.848269    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/577e3a93-0560-4099-bddb-dd55563d002d-original-pull-secret podName:577e3a93-0560-4099-bddb-dd55563d002d nodeName:}" failed. No retries permitted until 2026-07-12 17:03:14.848251676 +0000 UTC m=+2.968568457 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/577e3a93-0560-4099-bddb-dd55563d002d-original-pull-secret") pod "global-pull-secret-syncer-mkw2b" (UID: "577e3a93-0560-4099-bddb-dd55563d002d") : object "kube-system"/"original-pull-secret" not registered
Jul 12 17:03:13.848470 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:03:13.848439    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 12 17:03:13.848545 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:03:13.848485    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1d6c71b-d9fb-4f2a-9a81-5e8f0f8a6c13-metrics-certs podName:f1d6c71b-d9fb-4f2a-9a81-5e8f0f8a6c13 nodeName:}" failed. No retries permitted until 2026-07-12 17:03:14.848472058 +0000 UTC m=+2.968788845 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f1d6c71b-d9fb-4f2a-9a81-5e8f0f8a6c13-metrics-certs") pod "network-metrics-daemon-h442x" (UID: "f1d6c71b-d9fb-4f2a-9a81-5e8f0f8a6c13") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 12 17:03:13.949303 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:13.949272    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6sf9\" (UniqueName: \"kubernetes.io/projected/259ec8ed-10bd-4fae-8f7f-97409bd4f54b-kube-api-access-n6sf9\") pod \"network-check-target-gxtt8\" (UID: \"259ec8ed-10bd-4fae-8f7f-97409bd4f54b\") " pod="openshift-network-diagnostics/network-check-target-gxtt8"
Jul 12 17:03:13.949501 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:03:13.949450    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 12 17:03:13.949593 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:03:13.949518    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 12 17:03:13.949593 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:03:13.949550    2569 projected.go:196] Error preparing data for projected volume kube-api-access-n6sf9 for pod openshift-network-diagnostics/network-check-target-gxtt8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 12 17:03:13.949695 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:03:13.949632    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/259ec8ed-10bd-4fae-8f7f-97409bd4f54b-kube-api-access-n6sf9 podName:259ec8ed-10bd-4fae-8f7f-97409bd4f54b nodeName:}" failed. No retries permitted until 2026-07-12 17:03:14.949613937 +0000 UTC m=+3.069930728 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-n6sf9" (UniqueName: "kubernetes.io/projected/259ec8ed-10bd-4fae-8f7f-97409bd4f54b-kube-api-access-n6sf9") pod "network-check-target-gxtt8" (UID: "259ec8ed-10bd-4fae-8f7f-97409bd4f54b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 12 17:03:14.288440 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:14.288280    2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-11 16:58:13 +0000 UTC" deadline="2028-03-21 09:29:19.065589624 +0000 UTC"
Jul 12 17:03:14.288440 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:14.288337    2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14824h26m4.777256249s"
Jul 12 17:03:14.375219 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:14.374708    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gxtt8"
Jul 12 17:03:14.375219 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:03:14.374837    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gxtt8" podUID="259ec8ed-10bd-4fae-8f7f-97409bd4f54b"
Jul 12 17:03:14.399603 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:14.399545    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-nvmct" event={"ID":"8bcfc1cb-b938-4719-a485-dc3acd069a11","Type":"ContainerStarted","Data":"87fbf994de27e044c9671b311fe672939212decc4d0a3a5c8a2d955ddef4abab"}
Jul 12 17:03:14.405382 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:14.405352    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9xbrj" event={"ID":"6b618491-4e81-4708-ab1b-2781d1df7c87","Type":"ContainerStarted","Data":"ecce4def78cf504b75a53b58afb94dc17b6cb4edea8c651b5367681ce1bf4bc5"}
Jul 12 17:03:14.423394 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:14.423364    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-xsskc" event={"ID":"ecab5f05-562e-47bc-9c5a-04893ab8a903","Type":"ContainerStarted","Data":"59949c578e09968c95295ad84bdf9f6786db993d3427b0f884bdbf0d3ad6ff6a"}
Jul 12 17:03:14.430958 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:14.430931    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-kjdjh" event={"ID":"5598dd9a-714f-4e7d-9c59-a6e2f61c5a74","Type":"ContainerStarted","Data":"30eb2a74f28528aecdcf2b5bd817ddac927661fc48574d5e591519c5aadaec9e"}
Jul 12 17:03:14.451108 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:14.451077    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ltcld" event={"ID":"59055e87-8c03-4225-97e1-62114754af67","Type":"ContainerStarted","Data":"6c91253dc2d761ef2c80e0b4552c8e082513d4142327989f19da061346eca0fd"}
Jul 12 17:03:14.465542 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:14.465451    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-gpzg5" event={"ID":"098853ee-bad5-44d4-b2ac-bc4aa97b5d1c","Type":"ContainerStarted","Data":"46358fdf14c86d0970b57520cebb2cba4c2c4420d325124b3702d7a65e5a2d79"}
Jul 12 17:03:14.470176 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:14.470150    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-zlfcq" event={"ID":"fcdb5e54-4227-4aed-bcb8-b019fc5dfd86","Type":"ContainerStarted","Data":"bff536fc8dfc910c3e1f704aeb2ec9d171f1bbe6b0fb04beccf611e165512d22"}
Jul 12 17:03:14.476837 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:14.476643    2569 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 12 17:03:14.479062 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:14.478992    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-9bc6r" event={"ID":"c8fd63f2-13ee-4231-8ea3-a83dd90f365a","Type":"ContainerStarted","Data":"83e5abb67c537b44e43d064f7e2da0f097331f6ee66a09c893c1d7802bedea8b"}
Jul 12 17:03:14.493601 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:14.493555    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-smtm5" event={"ID":"8dc74bb2-e60b-49c3-9d2a-0bf12cdbc40a","Type":"ContainerStarted","Data":"3f2596709c056512fb7558c7b9d8fb591bc65f9c8314593c56d1762d899fb7c3"}
Jul 12 17:03:14.856647 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:14.856605    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/577e3a93-0560-4099-bddb-dd55563d002d-original-pull-secret\") pod \"global-pull-secret-syncer-mkw2b\" (UID: \"577e3a93-0560-4099-bddb-dd55563d002d\") " pod="kube-system/global-pull-secret-syncer-mkw2b"
Jul 12 17:03:14.856817 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:14.856695    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1d6c71b-d9fb-4f2a-9a81-5e8f0f8a6c13-metrics-certs\") pod \"network-metrics-daemon-h442x\" (UID: \"f1d6c71b-d9fb-4f2a-9a81-5e8f0f8a6c13\") " pod="openshift-multus/network-metrics-daemon-h442x"
Jul 12 17:03:14.856903 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:03:14.856845    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 12 17:03:14.856952 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:03:14.856916    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1d6c71b-d9fb-4f2a-9a81-5e8f0f8a6c13-metrics-certs podName:f1d6c71b-d9fb-4f2a-9a81-5e8f0f8a6c13 nodeName:}" failed. No retries permitted until 2026-07-12 17:03:16.856897594 +0000 UTC m=+4.977214374 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f1d6c71b-d9fb-4f2a-9a81-5e8f0f8a6c13-metrics-certs") pod "network-metrics-daemon-h442x" (UID: "f1d6c71b-d9fb-4f2a-9a81-5e8f0f8a6c13") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 12 17:03:14.857367 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:03:14.857348    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 12 17:03:14.857460 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:03:14.857406    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/577e3a93-0560-4099-bddb-dd55563d002d-original-pull-secret podName:577e3a93-0560-4099-bddb-dd55563d002d nodeName:}" failed. No retries permitted until 2026-07-12 17:03:16.857390589 +0000 UTC m=+4.977707364 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/577e3a93-0560-4099-bddb-dd55563d002d-original-pull-secret") pod "global-pull-secret-syncer-mkw2b" (UID: "577e3a93-0560-4099-bddb-dd55563d002d") : object "kube-system"/"original-pull-secret" not registered
Jul 12 17:03:14.930146 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:14.930117    2569 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 12 17:03:14.959593 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:14.957768    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6sf9\" (UniqueName: \"kubernetes.io/projected/259ec8ed-10bd-4fae-8f7f-97409bd4f54b-kube-api-access-n6sf9\") pod \"network-check-target-gxtt8\" (UID: \"259ec8ed-10bd-4fae-8f7f-97409bd4f54b\") " pod="openshift-network-diagnostics/network-check-target-gxtt8"
Jul 12 17:03:14.959593 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:03:14.957963    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 12 17:03:14.959593 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:03:14.957980    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 12 17:03:14.959593 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:03:14.957993    2569 projected.go:196] Error preparing data for projected volume kube-api-access-n6sf9 for pod openshift-network-diagnostics/network-check-target-gxtt8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 12 17:03:14.959593 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:03:14.958045    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/259ec8ed-10bd-4fae-8f7f-97409bd4f54b-kube-api-access-n6sf9 podName:259ec8ed-10bd-4fae-8f7f-97409bd4f54b nodeName:}" failed. No retries permitted until 2026-07-12 17:03:16.958028574 +0000 UTC m=+5.078345355 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-n6sf9" (UniqueName: "kubernetes.io/projected/259ec8ed-10bd-4fae-8f7f-97409bd4f54b-kube-api-access-n6sf9") pod "network-check-target-gxtt8" (UID: "259ec8ed-10bd-4fae-8f7f-97409bd4f54b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 12 17:03:15.289059 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:15.288932    2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-11 16:58:13 +0000 UTC" deadline="2027-12-10 13:54:41.437831305 +0000 UTC"
Jul 12 17:03:15.289059 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:15.288998    2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12380h51m26.148837348s"
Jul 12 17:03:15.373618 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:15.373585    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mkw2b"
Jul 12 17:03:15.373799 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:03:15.373715    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mkw2b" podUID="577e3a93-0560-4099-bddb-dd55563d002d"
Jul 12 17:03:15.374138 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:15.374087    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h442x"
Jul 12 17:03:15.374280 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:03:15.374201    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h442x" podUID="f1d6c71b-d9fb-4f2a-9a81-5e8f0f8a6c13"
Jul 12 17:03:16.375376 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:16.374907    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gxtt8"
Jul 12 17:03:16.375376 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:03:16.375029    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gxtt8" podUID="259ec8ed-10bd-4fae-8f7f-97409bd4f54b"
Jul 12 17:03:16.880039 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:16.880011    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1d6c71b-d9fb-4f2a-9a81-5e8f0f8a6c13-metrics-certs\") pod \"network-metrics-daemon-h442x\" (UID: \"f1d6c71b-d9fb-4f2a-9a81-5e8f0f8a6c13\") " pod="openshift-multus/network-metrics-daemon-h442x"
Jul 12 17:03:16.880192 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:16.880077    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/577e3a93-0560-4099-bddb-dd55563d002d-original-pull-secret\") pod \"global-pull-secret-syncer-mkw2b\" (UID: \"577e3a93-0560-4099-bddb-dd55563d002d\") " pod="kube-system/global-pull-secret-syncer-mkw2b"
Jul 12 17:03:16.880192 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:03:16.880176    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 12 17:03:16.880311 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:03:16.880197    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 12 17:03:16.880311 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:03:16.880247    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1d6c71b-d9fb-4f2a-9a81-5e8f0f8a6c13-metrics-certs podName:f1d6c71b-d9fb-4f2a-9a81-5e8f0f8a6c13 nodeName:}" failed. No retries permitted until 2026-07-12 17:03:20.880226648 +0000 UTC m=+9.000543425 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f1d6c71b-d9fb-4f2a-9a81-5e8f0f8a6c13-metrics-certs") pod "network-metrics-daemon-h442x" (UID: "f1d6c71b-d9fb-4f2a-9a81-5e8f0f8a6c13") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 12 17:03:16.880311 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:03:16.880267    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/577e3a93-0560-4099-bddb-dd55563d002d-original-pull-secret podName:577e3a93-0560-4099-bddb-dd55563d002d nodeName:}" failed. No retries permitted until 2026-07-12 17:03:20.880257296 +0000 UTC m=+9.000574071 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/577e3a93-0560-4099-bddb-dd55563d002d-original-pull-secret") pod "global-pull-secret-syncer-mkw2b" (UID: "577e3a93-0560-4099-bddb-dd55563d002d") : object "kube-system"/"original-pull-secret" not registered
Jul 12 17:03:16.981048 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:16.980932    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6sf9\" (UniqueName: \"kubernetes.io/projected/259ec8ed-10bd-4fae-8f7f-97409bd4f54b-kube-api-access-n6sf9\") pod \"network-check-target-gxtt8\" (UID: \"259ec8ed-10bd-4fae-8f7f-97409bd4f54b\") " pod="openshift-network-diagnostics/network-check-target-gxtt8"
Jul 12 17:03:16.981224 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:03:16.981094    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 12 17:03:16.981224 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:03:16.981118    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 12 17:03:16.981224 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:03:16.981129    2569 projected.go:196] Error preparing data for projected volume kube-api-access-n6sf9 for pod openshift-network-diagnostics/network-check-target-gxtt8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 12 17:03:16.981224 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:03:16.981181    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/259ec8ed-10bd-4fae-8f7f-97409bd4f54b-kube-api-access-n6sf9 podName:259ec8ed-10bd-4fae-8f7f-97409bd4f54b nodeName:}" failed. No retries permitted until 2026-07-12 17:03:20.981163297 +0000 UTC m=+9.101480084 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-n6sf9" (UniqueName: "kubernetes.io/projected/259ec8ed-10bd-4fae-8f7f-97409bd4f54b-kube-api-access-n6sf9") pod "network-check-target-gxtt8" (UID: "259ec8ed-10bd-4fae-8f7f-97409bd4f54b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 12 17:03:17.373702 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:17.373621    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h442x"
Jul 12 17:03:17.373851 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:03:17.373772    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h442x" podUID="f1d6c71b-d9fb-4f2a-9a81-5e8f0f8a6c13"
Jul 12 17:03:17.373908 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:17.373853    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mkw2b"
Jul 12 17:03:17.374032 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:03:17.373969    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mkw2b" podUID="577e3a93-0560-4099-bddb-dd55563d002d"
Jul 12 17:03:18.373803 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:18.373770    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gxtt8"
Jul 12 17:03:18.374251 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:03:18.373908    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gxtt8" podUID="259ec8ed-10bd-4fae-8f7f-97409bd4f54b"
Jul 12 17:03:19.373708 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:19.373668    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mkw2b"
Jul 12 17:03:19.373880 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:19.373668    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h442x"
Jul 12 17:03:19.373880 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:03:19.373814    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mkw2b" podUID="577e3a93-0560-4099-bddb-dd55563d002d"
Jul 12 17:03:19.373880 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:03:19.373870    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h442x" podUID="f1d6c71b-d9fb-4f2a-9a81-5e8f0f8a6c13"
Jul 12 17:03:20.373726 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:20.373695    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gxtt8"
Jul 12 17:03:20.373899 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:03:20.373830    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gxtt8" podUID="259ec8ed-10bd-4fae-8f7f-97409bd4f54b"
Jul 12 17:03:20.912631 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:20.912454    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/577e3a93-0560-4099-bddb-dd55563d002d-original-pull-secret\") pod \"global-pull-secret-syncer-mkw2b\" (UID: \"577e3a93-0560-4099-bddb-dd55563d002d\") " pod="kube-system/global-pull-secret-syncer-mkw2b"
Jul 12 17:03:20.912631 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:20.912551    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1d6c71b-d9fb-4f2a-9a81-5e8f0f8a6c13-metrics-certs\") pod \"network-metrics-daemon-h442x\" (UID: \"f1d6c71b-d9fb-4f2a-9a81-5e8f0f8a6c13\") " pod="openshift-multus/network-metrics-daemon-h442x"
Jul 12 17:03:20.912866 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:03:20.912669    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 12 17:03:20.912866 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:03:20.912751    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/577e3a93-0560-4099-bddb-dd55563d002d-original-pull-secret podName:577e3a93-0560-4099-bddb-dd55563d002d nodeName:}" failed. No retries permitted until 2026-07-12 17:03:28.912728373 +0000 UTC m=+17.033045152 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/577e3a93-0560-4099-bddb-dd55563d002d-original-pull-secret") pod "global-pull-secret-syncer-mkw2b" (UID: "577e3a93-0560-4099-bddb-dd55563d002d") : object "kube-system"/"original-pull-secret" not registered
Jul 12 17:03:20.913111 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:03:20.913029    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 12 17:03:20.913111 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:03:20.913093    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1d6c71b-d9fb-4f2a-9a81-5e8f0f8a6c13-metrics-certs podName:f1d6c71b-d9fb-4f2a-9a81-5e8f0f8a6c13 nodeName:}" failed. No retries permitted until 2026-07-12 17:03:28.913073767 +0000 UTC m=+17.033390548 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f1d6c71b-d9fb-4f2a-9a81-5e8f0f8a6c13-metrics-certs") pod "network-metrics-daemon-h442x" (UID: "f1d6c71b-d9fb-4f2a-9a81-5e8f0f8a6c13") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 12 17:03:21.012858 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:21.012823    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6sf9\" (UniqueName: \"kubernetes.io/projected/259ec8ed-10bd-4fae-8f7f-97409bd4f54b-kube-api-access-n6sf9\") pod \"network-check-target-gxtt8\" (UID: \"259ec8ed-10bd-4fae-8f7f-97409bd4f54b\") " pod="openshift-network-diagnostics/network-check-target-gxtt8"
Jul 12 17:03:21.013054 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:03:21.013035    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 12 17:03:21.013131 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:03:21.013058    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 12 17:03:21.013131 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:03:21.013068    2569 projected.go:196] Error preparing data for projected volume kube-api-access-n6sf9 for pod openshift-network-diagnostics/network-check-target-gxtt8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 12 17:03:21.013131 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:03:21.013118    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/259ec8ed-10bd-4fae-8f7f-97409bd4f54b-kube-api-access-n6sf9 podName:259ec8ed-10bd-4fae-8f7f-97409bd4f54b nodeName:}" failed. No retries permitted until 2026-07-12 17:03:29.013101193 +0000 UTC m=+17.133417982 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-n6sf9" (UniqueName: "kubernetes.io/projected/259ec8ed-10bd-4fae-8f7f-97409bd4f54b-kube-api-access-n6sf9") pod "network-check-target-gxtt8" (UID: "259ec8ed-10bd-4fae-8f7f-97409bd4f54b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 12 17:03:21.373556 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:21.373418    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h442x"
Jul 12 17:03:21.373556 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:03:21.373535    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h442x" podUID="f1d6c71b-d9fb-4f2a-9a81-5e8f0f8a6c13"
Jul 12 17:03:21.373811 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:21.373796    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mkw2b"
Jul 12 17:03:21.373880 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:03:21.373865    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mkw2b" podUID="577e3a93-0560-4099-bddb-dd55563d002d"
Jul 12 17:03:22.375055 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:22.374684    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gxtt8"
Jul 12 17:03:22.375055 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:03:22.374777    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gxtt8" podUID="259ec8ed-10bd-4fae-8f7f-97409bd4f54b"
Jul 12 17:03:23.373976 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:23.373860    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mkw2b"
Jul 12 17:03:23.374154 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:03:23.373982    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mkw2b" podUID="577e3a93-0560-4099-bddb-dd55563d002d"
Jul 12 17:03:23.374154 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:23.374033    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h442x"
Jul 12 17:03:23.374154 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:03:23.374135    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h442x" podUID="f1d6c71b-d9fb-4f2a-9a81-5e8f0f8a6c13"
Jul 12 17:03:24.374022 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:24.373986    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gxtt8"
Jul 12 17:03:24.374443 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:03:24.374121    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gxtt8" podUID="259ec8ed-10bd-4fae-8f7f-97409bd4f54b"
Jul 12 17:03:25.373886 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:25.373853    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mkw2b"
Jul 12 17:03:25.374060 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:03:25.373983    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mkw2b" podUID="577e3a93-0560-4099-bddb-dd55563d002d"
Jul 12 17:03:25.374060 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:25.374037    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h442x"
Jul 12 17:03:25.374457 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:03:25.374136    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h442x" podUID="f1d6c71b-d9fb-4f2a-9a81-5e8f0f8a6c13"
Jul 12 17:03:26.373601 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:26.373568    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gxtt8"
Jul 12 17:03:26.373780 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:03:26.373675    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gxtt8" podUID="259ec8ed-10bd-4fae-8f7f-97409bd4f54b"
Jul 12 17:03:27.373460 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:27.373426    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mkw2b"
Jul 12 17:03:27.373460 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:27.373436    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h442x"
Jul 12 17:03:27.373956 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:03:27.373565    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mkw2b" podUID="577e3a93-0560-4099-bddb-dd55563d002d"
Jul 12 17:03:27.373956 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:03:27.373718    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h442x" podUID="f1d6c71b-d9fb-4f2a-9a81-5e8f0f8a6c13"
Jul 12 17:03:28.374114 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:28.374080    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gxtt8"
Jul 12 17:03:28.374593 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:03:28.374208    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gxtt8" podUID="259ec8ed-10bd-4fae-8f7f-97409bd4f54b"
Jul 12 17:03:28.974961 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:28.974870    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1d6c71b-d9fb-4f2a-9a81-5e8f0f8a6c13-metrics-certs\") pod \"network-metrics-daemon-h442x\" (UID: \"f1d6c71b-d9fb-4f2a-9a81-5e8f0f8a6c13\") " pod="openshift-multus/network-metrics-daemon-h442x"
Jul 12 17:03:28.975128 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:03:28.975022    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 12 17:03:28.975128 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:03:28.975078    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1d6c71b-d9fb-4f2a-9a81-5e8f0f8a6c13-metrics-certs podName:f1d6c71b-d9fb-4f2a-9a81-5e8f0f8a6c13 nodeName:}" failed. No retries permitted until 2026-07-12 17:03:44.975064079 +0000 UTC m=+33.095380854 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f1d6c71b-d9fb-4f2a-9a81-5e8f0f8a6c13-metrics-certs") pod "network-metrics-daemon-h442x" (UID: "f1d6c71b-d9fb-4f2a-9a81-5e8f0f8a6c13") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 12 17:03:28.975128 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:28.975029    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/577e3a93-0560-4099-bddb-dd55563d002d-original-pull-secret\") pod \"global-pull-secret-syncer-mkw2b\" (UID: \"577e3a93-0560-4099-bddb-dd55563d002d\") " pod="kube-system/global-pull-secret-syncer-mkw2b"
Jul 12 17:03:28.975128 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:03:28.975096    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 12 17:03:28.975266 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:03:28.975152    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/577e3a93-0560-4099-bddb-dd55563d002d-original-pull-secret podName:577e3a93-0560-4099-bddb-dd55563d002d nodeName:}" failed. No retries permitted until 2026-07-12 17:03:44.975136432 +0000 UTC m=+33.095453207 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/577e3a93-0560-4099-bddb-dd55563d002d-original-pull-secret") pod "global-pull-secret-syncer-mkw2b" (UID: "577e3a93-0560-4099-bddb-dd55563d002d") : object "kube-system"/"original-pull-secret" not registered
Jul 12 17:03:29.075867 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:29.075831    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6sf9\" (UniqueName: \"kubernetes.io/projected/259ec8ed-10bd-4fae-8f7f-97409bd4f54b-kube-api-access-n6sf9\") pod \"network-check-target-gxtt8\" (UID: \"259ec8ed-10bd-4fae-8f7f-97409bd4f54b\") " pod="openshift-network-diagnostics/network-check-target-gxtt8"
Jul 12 17:03:29.076042 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:03:29.076017    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 12 17:03:29.076090 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:03:29.076045    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 12 17:03:29.076090 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:03:29.076059    2569 projected.go:196] Error preparing data for projected volume kube-api-access-n6sf9 for pod openshift-network-diagnostics/network-check-target-gxtt8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 12 17:03:29.076179 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:03:29.076118    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/259ec8ed-10bd-4fae-8f7f-97409bd4f54b-kube-api-access-n6sf9 podName:259ec8ed-10bd-4fae-8f7f-97409bd4f54b nodeName:}" failed. No retries permitted until 2026-07-12 17:03:45.076099257 +0000 UTC m=+33.196416045 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-n6sf9" (UniqueName: "kubernetes.io/projected/259ec8ed-10bd-4fae-8f7f-97409bd4f54b-kube-api-access-n6sf9") pod "network-check-target-gxtt8" (UID: "259ec8ed-10bd-4fae-8f7f-97409bd4f54b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 12 17:03:29.373892 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:29.373819    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mkw2b"
Jul 12 17:03:29.374064 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:03:29.373934    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mkw2b" podUID="577e3a93-0560-4099-bddb-dd55563d002d"
Jul 12 17:03:29.374064 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:29.373989    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h442x"
Jul 12 17:03:29.374176 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:03:29.374117    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h442x" podUID="f1d6c71b-d9fb-4f2a-9a81-5e8f0f8a6c13"
Jul 12 17:03:30.373965 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:30.373932    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gxtt8"
Jul 12 17:03:30.374223 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:03:30.374065    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gxtt8" podUID="259ec8ed-10bd-4fae-8f7f-97409bd4f54b"
Jul 12 17:03:31.374428 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:31.374386    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h442x"
Jul 12 17:03:31.374911 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:31.374442    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mkw2b"
Jul 12 17:03:31.374911 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:03:31.374544    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h442x" podUID="f1d6c71b-d9fb-4f2a-9a81-5e8f0f8a6c13"
Jul 12 17:03:31.374911 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:03:31.374653    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mkw2b" podUID="577e3a93-0560-4099-bddb-dd55563d002d"
Jul 12 17:03:32.374378 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:32.374342    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gxtt8"
Jul 12 17:03:32.374563 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:03:32.374450    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gxtt8" podUID="259ec8ed-10bd-4fae-8f7f-97409bd4f54b"
Jul 12 17:03:33.373864 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:33.373842    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mkw2b"
Jul 12 17:03:33.373987 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:03:33.373935    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mkw2b" podUID="577e3a93-0560-4099-bddb-dd55563d002d"
Jul 12 17:03:33.374034 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:33.374008    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h442x"
Jul 12 17:03:33.374121 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:03:33.374105    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h442x" podUID="f1d6c71b-d9fb-4f2a-9a81-5e8f0f8a6c13"
Jul 12 17:03:33.544019 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:33.543886    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-94.ec2.internal" event={"ID":"64598ecc4535f6658a01b7cab4f3caeb","Type":"ContainerStarted","Data":"a9e0b9c2882a887d4b630ac50f37cde44cffb2f6e40aea9158b1e79a62e96c06"}
Jul 12 17:03:33.545452 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:33.545426    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-nvmct" event={"ID":"8bcfc1cb-b938-4719-a485-dc3acd069a11","Type":"ContainerStarted","Data":"2b55f8f781aebd824a5bab8b27ff7dcc37538d887631cd66e7cf5d7b4170f970"}
Jul 12 17:03:33.546947 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:33.546928    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9xbrj" event={"ID":"6b618491-4e81-4708-ab1b-2781d1df7c87","Type":"ContainerStarted","Data":"fdd6aa83156cfe1b0c6f850771e59f0d70e3a7e89bb96d8a6e9a73b72758b123"}
Jul 12 17:03:33.586200 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:33.586157    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-nvmct" podStartSLOduration=1.9096004500000001 podStartE2EDuration="21.586141608s" totalImagesPullingTime="19.676541158s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-12 17:03:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 17:03:33.585155486 +0000 UTC m=+21.705472284" watchObservedRunningTime="2026-07-12 17:03:33.586141608 +0000 UTC m=+21.706458405"
Jul 12 17:03:33.586568 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:33.586487    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-94.ec2.internal" podStartSLOduration=20.586477704 podStartE2EDuration="20.586477704s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-12 17:03:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 17:03:33.560453723 +0000 UTC m=+21.680770520" watchObservedRunningTime="2026-07-12 17:03:33.586477704 +0000 UTC m=+21.706794500"
Jul 12 17:03:34.373566 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:34.373328    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gxtt8"
Jul 12 17:03:34.373739 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:03:34.373600    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gxtt8" podUID="259ec8ed-10bd-4fae-8f7f-97409bd4f54b"
Jul 12 17:03:34.551231 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:34.551196    2569 generic.go:356] "Generic (PLEG): container finished" podID="59055e87-8c03-4225-97e1-62114754af67" containerID="4fce0b9f3bc2043114fa0d5277a464ea592200491ae45f782f31bad86dcbf743" exitCode=0
Jul 12 17:03:34.552083 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:34.551313    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ltcld" event={"ID":"59055e87-8c03-4225-97e1-62114754af67","Type":"ContainerDied","Data":"4fce0b9f3bc2043114fa0d5277a464ea592200491ae45f782f31bad86dcbf743"}
Jul 12 17:03:34.553038 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:34.553014    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-gpzg5" event={"ID":"098853ee-bad5-44d4-b2ac-bc4aa97b5d1c","Type":"ContainerStarted","Data":"316cc1cc18f8a8e36bab12b882f367f7536661450daecd832862f1b50be423df"}
Jul 12 17:03:34.554734 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:34.554709    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-zlfcq" event={"ID":"fcdb5e54-4227-4aed-bcb8-b019fc5dfd86","Type":"ContainerStarted","Data":"0de6df1c8140c84882f4c9e86b969a02c6dc4c8f09f37d1cd195abe5799598c7"}
Jul 12 17:03:34.556524 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:34.556491    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-smtm5" event={"ID":"8dc74bb2-e60b-49c3-9d2a-0bf12cdbc40a","Type":"ContainerStarted","Data":"bfeea1c26495aff4d71bed87f972db4cdc5d494ee641098f02d0c2d51cde4332"}
Jul 12 17:03:34.562282 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:34.562259    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9xbrj_6b618491-4e81-4708-ab1b-2781d1df7c87/ovn-acl-logging/0.log"
Jul 12 17:03:34.562670 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:34.562644    2569 generic.go:356] "Generic (PLEG): container finished" podID="6b618491-4e81-4708-ab1b-2781d1df7c87" containerID="10ed43b8693c1c33730295c1d78b30799cb4f6b118c3dfa706af9a3436f35d8f" exitCode=1
Jul 12 17:03:34.562767 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:34.562679    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9xbrj" event={"ID":"6b618491-4e81-4708-ab1b-2781d1df7c87","Type":"ContainerStarted","Data":"d57a6dc26423639ec0462d61b99355da27afe87d802fb85f5b10c25730820ad2"}
Jul 12 17:03:34.562836 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:34.562796    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9xbrj" event={"ID":"6b618491-4e81-4708-ab1b-2781d1df7c87","Type":"ContainerStarted","Data":"aaf756ee4cc7badf2cc72da6408edb84ce65123beb27f7c1a617c6607c66f515"}
Jul 12 17:03:34.562891 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:34.562853    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9xbrj" event={"ID":"6b618491-4e81-4708-ab1b-2781d1df7c87","Type":"ContainerStarted","Data":"42e362110c8b68c49540b518c4ed3a3aee22721874aa037fafe7fce79ce84ac5"}
Jul 12 17:03:34.562944 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:34.562899    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9xbrj" event={"ID":"6b618491-4e81-4708-ab1b-2781d1df7c87","Type":"ContainerStarted","Data":"6bd74347cb77beb332434e7ace7e5aba09c0b95d86d86f19f558fc7deaca4f87"}
Jul 12 17:03:34.562996 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:34.562942    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9xbrj" event={"ID":"6b618491-4e81-4708-ab1b-2781d1df7c87","Type":"ContainerDied","Data":"10ed43b8693c1c33730295c1d78b30799cb4f6b118c3dfa706af9a3436f35d8f"}
Jul 12 17:03:34.564361 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:34.564334    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-xsskc" event={"ID":"ecab5f05-562e-47bc-9c5a-04893ab8a903","Type":"ContainerStarted","Data":"f97e2be4d8271361ae3a7bed130da7014c705fd6560159b5df4aee40f6d2bfe1"}
Jul 12 17:03:34.565859 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:34.565834    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-kjdjh" event={"ID":"5598dd9a-714f-4e7d-9c59-a6e2f61c5a74","Type":"ContainerStarted","Data":"cfb9725df65f5411820ccb35d38f5cb9d39537e6bdd19c9a4bd1e2fc5dbce7d7"}
Jul 12 17:03:34.567640 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:34.567615    2569 generic.go:356] "Generic (PLEG): container finished" podID="bf9d9cd767ec6f08a024f6a8e7fbc81d" containerID="8a17553060f39a6c8b41e294c41ded005b3f5a68b6803cb20ab00cf13187dc61" exitCode=0
Jul 12 17:03:34.567746 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:34.567695    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-94.ec2.internal" event={"ID":"bf9d9cd767ec6f08a024f6a8e7fbc81d","Type":"ContainerDied","Data":"8a17553060f39a6c8b41e294c41ded005b3f5a68b6803cb20ab00cf13187dc61"}
Jul 12 17:03:34.600927 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:34.600877    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-gpzg5" podStartSLOduration=11.319802161 podStartE2EDuration="22.600862945s" totalImagesPullingTime="11.281060784s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-12 17:03:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 17:03:34.600454813 +0000 UTC m=+22.720771606" watchObservedRunningTime="2026-07-12 17:03:34.600862945 +0000 UTC m=+22.721179742"
Jul 12 17:03:34.637326 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:34.637197    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-kjdjh" podStartSLOduration=2.962864214 podStartE2EDuration="22.637176278s" totalImagesPullingTime="19.674312064s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 17:03:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 17:03:34.636810736 +0000 UTC m=+22.757127531" watchObservedRunningTime="2026-07-12 17:03:34.637176278 +0000 UTC m=+22.757493073"
Jul 12 17:03:34.654915 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:34.654827    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-xsskc" podStartSLOduration=2.687936023 podStartE2EDuration="22.654803937s" totalImagesPullingTime="19.966867914s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-12 17:03:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 17:03:34.653978134 +0000 UTC m=+22.774294929" watchObservedRunningTime="2026-07-12 17:03:34.654803937 +0000 UTC m=+22.775120736"
Jul 12 17:03:34.669954 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:34.669904    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-zlfcq" podStartSLOduration=3.030343936 podStartE2EDuration="22.66988396s" totalImagesPullingTime="19.639540024s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-12 17:03:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 17:03:34.669212495 +0000 UTC m=+22.789529293" watchObservedRunningTime="2026-07-12 17:03:34.66988396 +0000 UTC m=+22.790200754"
Jul 12 17:03:35.065533 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:35.065339    2569 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 12 17:03:35.303829 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:35.303661    2569 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-12T17:03:35.065500321Z","UUID":"4f1eddc8-ea42-4046-8af7-3333453d1e5d","Handler":null,"Name":"","Endpoint":""}
Jul 12 17:03:35.305310 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:35.305283    2569 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 12 17:03:35.305437 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:35.305326    2569 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 12 17:03:35.374376 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:35.374340    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h442x"
Jul 12 17:03:35.374376 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:35.374374    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mkw2b"
Jul 12 17:03:35.374676 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:03:35.374469    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h442x" podUID="f1d6c71b-d9fb-4f2a-9a81-5e8f0f8a6c13"
Jul 12 17:03:35.374676 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:03:35.374621    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mkw2b" podUID="577e3a93-0560-4099-bddb-dd55563d002d"
Jul 12 17:03:35.572918 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:35.572821    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-smtm5" event={"ID":"8dc74bb2-e60b-49c3-9d2a-0bf12cdbc40a","Type":"ContainerStarted","Data":"110d1aa12c83c8d1717b1bc81a47a33d8f07d5df26242422d63caa249e6225ed"}
Jul 12 17:03:35.575255 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:35.575224    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-94.ec2.internal" event={"ID":"bf9d9cd767ec6f08a024f6a8e7fbc81d","Type":"ContainerStarted","Data":"d69b04a89b0a094a50e92b2c6583d247620d40e2f3c79ca0d19423f57e2cdfa6"}
Jul 12 17:03:35.577679 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:35.576930    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-9bc6r" event={"ID":"c8fd63f2-13ee-4231-8ea3-a83dd90f365a","Type":"ContainerStarted","Data":"791568213a8a808fdb47e96357c661fbddb67695391262f2b92c9405324e120f"}
Jul 12 17:03:35.609068 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:35.609018    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-94.ec2.internal" podStartSLOduration=22.609001399 podStartE2EDuration="22.609001399s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-12 17:03:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 17:03:35.608502763 +0000 UTC m=+23.728819561" watchObservedRunningTime="2026-07-12 17:03:35.609001399 +0000 UTC m=+23.729318197"
Jul 12 17:03:35.623869 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:35.623824    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-9bc6r" podStartSLOduration=3.920815933 podStartE2EDuration="23.623809515s" totalImagesPullingTime="19.702993582s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-12 17:03:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 17:03:35.623554708 +0000 UTC m=+23.743871506" watchObservedRunningTime="2026-07-12 17:03:35.623809515 +0000 UTC m=+23.744126311"
Jul 12 17:03:36.374122 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:36.374047    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gxtt8"
Jul 12 17:03:36.374354 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:03:36.374173    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gxtt8" podUID="259ec8ed-10bd-4fae-8f7f-97409bd4f54b"
Jul 12 17:03:36.581947 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:36.581912    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-smtm5" event={"ID":"8dc74bb2-e60b-49c3-9d2a-0bf12cdbc40a","Type":"ContainerStarted","Data":"4d3cd42ca67b03742205aaa8209e653e8a7c32217293028c12debc9c6717eae4"}
Jul 12 17:03:36.585583 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:36.585560    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9xbrj_6b618491-4e81-4708-ab1b-2781d1df7c87/ovn-acl-logging/0.log"
Jul 12 17:03:36.586143 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:36.586102    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9xbrj" event={"ID":"6b618491-4e81-4708-ab1b-2781d1df7c87","Type":"ContainerStarted","Data":"a61690d5fb5aa2b3528b70f4354dcad5ba06acf1f3a28e6119bb4c39ee664042"}
Jul 12 17:03:36.601686 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:36.601635    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-smtm5" podStartSLOduration=7.380999998 podStartE2EDuration="24.601620114s" totalImagesPullingTime="17.220620116s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-12 17:03:12 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 17:03:36.600974914 +0000 UTC m=+24.721291711" watchObservedRunningTime="2026-07-12 17:03:36.601620114 +0000 UTC m=+24.721936946"
Jul 12 17:03:37.373518 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:37.373466    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h442x"
Jul 12 17:03:37.373518 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:37.373497    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mkw2b"
Jul 12 17:03:37.373771 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:03:37.373615    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h442x" podUID="f1d6c71b-d9fb-4f2a-9a81-5e8f0f8a6c13"
Jul 12 17:03:37.373771 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:03:37.373752    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mkw2b" podUID="577e3a93-0560-4099-bddb-dd55563d002d"
Jul 12 17:03:37.803756 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:37.803716    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-kjdjh"
Jul 12 17:03:37.804374 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:37.804256    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-kjdjh"
Jul 12 17:03:38.373558 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:38.373500    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gxtt8"
Jul 12 17:03:38.373721 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:03:38.373662    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gxtt8" podUID="259ec8ed-10bd-4fae-8f7f-97409bd4f54b"
Jul 12 17:03:38.592015 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:38.591933    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-kjdjh"
Jul 12 17:03:38.592490 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:38.592470    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-kjdjh"
Jul 12 17:03:39.373919 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:39.373740    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mkw2b"
Jul 12 17:03:39.374488 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:39.373740    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h442x"
Jul 12 17:03:39.374488 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:03:39.374000    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mkw2b" podUID="577e3a93-0560-4099-bddb-dd55563d002d"
Jul 12 17:03:39.374488 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:03:39.374072    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h442x" podUID="f1d6c71b-d9fb-4f2a-9a81-5e8f0f8a6c13"
Jul 12 17:03:39.596929 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:39.596900    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9xbrj_6b618491-4e81-4708-ab1b-2781d1df7c87/ovn-acl-logging/0.log"
Jul 12 17:03:39.597274 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:39.597252    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9xbrj" event={"ID":"6b618491-4e81-4708-ab1b-2781d1df7c87","Type":"ContainerStarted","Data":"c49e341de8bb5cc0a12ddcf592a1193c8d23fd8a2b077412e765613f50b68b5d"}
Jul 12 17:03:39.597546 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:39.597500    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9xbrj"
Jul 12 17:03:39.597667 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:39.597589    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9xbrj"
Jul 12 17:03:39.597720 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:39.597684    2569 scope.go:117] "RemoveContainer" containerID="10ed43b8693c1c33730295c1d78b30799cb4f6b118c3dfa706af9a3436f35d8f"
Jul 12 17:03:39.599467 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:39.599442    2569 generic.go:356] "Generic (PLEG): container finished" podID="59055e87-8c03-4225-97e1-62114754af67" containerID="229c599ce1eebd06fdb705f347d00edc18342fe045d9c46bda638c056079cf0c" exitCode=0
Jul 12 17:03:39.599632 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:39.599613    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ltcld" event={"ID":"59055e87-8c03-4225-97e1-62114754af67","Type":"ContainerDied","Data":"229c599ce1eebd06fdb705f347d00edc18342fe045d9c46bda638c056079cf0c"}
Jul 12 17:03:39.614919 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:39.614900    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9xbrj"
Jul 12 17:03:40.373756 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:40.373577    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gxtt8"
Jul 12 17:03:40.373939 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:03:40.373835    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gxtt8" podUID="259ec8ed-10bd-4fae-8f7f-97409bd4f54b"
Jul 12 17:03:40.605041 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:40.605020    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9xbrj_6b618491-4e81-4708-ab1b-2781d1df7c87/ovn-acl-logging/0.log"
Jul 12 17:03:40.605347 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:40.605325    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9xbrj" event={"ID":"6b618491-4e81-4708-ab1b-2781d1df7c87","Type":"ContainerStarted","Data":"c69ad28cab5e1740275b757eee417306f50fc0b3097cbbd068d5e4fa7730447b"}
Jul 12 17:03:40.605675 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:40.605655    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9xbrj"
Jul 12 17:03:40.622685 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:40.622660    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9xbrj"
Jul 12 17:03:40.678480 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:40.678436    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-9xbrj" podStartSLOduration=8.938288831 podStartE2EDuration="28.678421517s" totalImagesPullingTime="19.740132686s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-12 17:03:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 17:03:40.642472661 +0000 UTC m=+28.762789460" watchObservedRunningTime="2026-07-12 17:03:40.678421517 +0000 UTC m=+28.798738317"
Jul 12 17:03:41.374246 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:41.374211    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h442x"
Jul 12 17:03:41.374246 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:41.374236    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mkw2b"
Jul 12 17:03:41.374659 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:03:41.374330    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mkw2b" podUID="577e3a93-0560-4099-bddb-dd55563d002d"
Jul 12 17:03:41.374659 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:03:41.374420    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h442x" podUID="f1d6c71b-d9fb-4f2a-9a81-5e8f0f8a6c13"
Jul 12 17:03:41.609485 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:41.609455    2569 generic.go:356] "Generic (PLEG): container finished" podID="59055e87-8c03-4225-97e1-62114754af67" containerID="bc0c0f23fbb14dff0d1f13fcb3a3990ec1fbeeec2c1c06bb24efabba51a225c1" exitCode=0
Jul 12 17:03:41.609637 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:41.609541    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ltcld" event={"ID":"59055e87-8c03-4225-97e1-62114754af67","Type":"ContainerDied","Data":"bc0c0f23fbb14dff0d1f13fcb3a3990ec1fbeeec2c1c06bb24efabba51a225c1"}
Jul 12 17:03:42.376216 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:42.376178    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gxtt8"
Jul 12 17:03:42.376931 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:03:42.376898    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gxtt8" podUID="259ec8ed-10bd-4fae-8f7f-97409bd4f54b"
Jul 12 17:03:42.616872 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:42.616841    2569 generic.go:356] "Generic (PLEG): container finished" podID="59055e87-8c03-4225-97e1-62114754af67" containerID="0870e7a12ecf9798377dd8f9ef8b6f6ba9cdb5a3b37c447c8457fd8eff21d6f9" exitCode=0
Jul 12 17:03:42.616954 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:42.616920    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ltcld" event={"ID":"59055e87-8c03-4225-97e1-62114754af67","Type":"ContainerDied","Data":"0870e7a12ecf9798377dd8f9ef8b6f6ba9cdb5a3b37c447c8457fd8eff21d6f9"}
Jul 12 17:03:43.374060 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:43.374021    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h442x"
Jul 12 17:03:43.374238 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:43.374065    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mkw2b"
Jul 12 17:03:43.374238 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:03:43.374152    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h442x" podUID="f1d6c71b-d9fb-4f2a-9a81-5e8f0f8a6c13"
Jul 12 17:03:43.374320 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:03:43.374284    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mkw2b" podUID="577e3a93-0560-4099-bddb-dd55563d002d"
Jul 12 17:03:44.377136 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:44.377102    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gxtt8"
Jul 12 17:03:44.377615 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:03:44.377205    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gxtt8" podUID="259ec8ed-10bd-4fae-8f7f-97409bd4f54b"
Jul 12 17:03:44.985654 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:44.985626    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/577e3a93-0560-4099-bddb-dd55563d002d-original-pull-secret\") pod \"global-pull-secret-syncer-mkw2b\" (UID: \"577e3a93-0560-4099-bddb-dd55563d002d\") " pod="kube-system/global-pull-secret-syncer-mkw2b"
Jul 12 17:03:44.985817 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:44.985681    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1d6c71b-d9fb-4f2a-9a81-5e8f0f8a6c13-metrics-certs\") pod \"network-metrics-daemon-h442x\" (UID: \"f1d6c71b-d9fb-4f2a-9a81-5e8f0f8a6c13\") " pod="openshift-multus/network-metrics-daemon-h442x"
Jul 12 17:03:44.985817 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:03:44.985772    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 12 17:03:44.985903 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:03:44.985833    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/577e3a93-0560-4099-bddb-dd55563d002d-original-pull-secret podName:577e3a93-0560-4099-bddb-dd55563d002d nodeName:}" failed. No retries permitted until 2026-07-12 17:04:16.985818797 +0000 UTC m=+65.106135572 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/577e3a93-0560-4099-bddb-dd55563d002d-original-pull-secret") pod "global-pull-secret-syncer-mkw2b" (UID: "577e3a93-0560-4099-bddb-dd55563d002d") : object "kube-system"/"original-pull-secret" not registered
Jul 12 17:03:44.985903 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:03:44.985775    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 12 17:03:44.985903 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:03:44.985902    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1d6c71b-d9fb-4f2a-9a81-5e8f0f8a6c13-metrics-certs podName:f1d6c71b-d9fb-4f2a-9a81-5e8f0f8a6c13 nodeName:}" failed. No retries permitted until 2026-07-12 17:04:16.985890704 +0000 UTC m=+65.106207479 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f1d6c71b-d9fb-4f2a-9a81-5e8f0f8a6c13-metrics-certs") pod "network-metrics-daemon-h442x" (UID: "f1d6c71b-d9fb-4f2a-9a81-5e8f0f8a6c13") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 12 17:03:45.086029 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:45.085990    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6sf9\" (UniqueName: \"kubernetes.io/projected/259ec8ed-10bd-4fae-8f7f-97409bd4f54b-kube-api-access-n6sf9\") pod \"network-check-target-gxtt8\" (UID: \"259ec8ed-10bd-4fae-8f7f-97409bd4f54b\") " pod="openshift-network-diagnostics/network-check-target-gxtt8"
Jul 12 17:03:45.086205 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:03:45.086127    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 12 17:03:45.086205 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:03:45.086157    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 12 17:03:45.086205 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:03:45.086172    2569 projected.go:196] Error preparing data for projected volume kube-api-access-n6sf9 for pod openshift-network-diagnostics/network-check-target-gxtt8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 12 17:03:45.086315 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:03:45.086237    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/259ec8ed-10bd-4fae-8f7f-97409bd4f54b-kube-api-access-n6sf9 podName:259ec8ed-10bd-4fae-8f7f-97409bd4f54b nodeName:}" failed. No retries permitted until 2026-07-12 17:04:17.086218795 +0000 UTC m=+65.206535575 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-n6sf9" (UniqueName: "kubernetes.io/projected/259ec8ed-10bd-4fae-8f7f-97409bd4f54b-kube-api-access-n6sf9") pod "network-check-target-gxtt8" (UID: "259ec8ed-10bd-4fae-8f7f-97409bd4f54b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 12 17:03:45.373459 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:45.373376    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h442x"
Jul 12 17:03:45.373629 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:45.373376    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mkw2b"
Jul 12 17:03:45.373629 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:03:45.373492    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h442x" podUID="f1d6c71b-d9fb-4f2a-9a81-5e8f0f8a6c13"
Jul 12 17:03:45.373629 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:03:45.373594    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mkw2b" podUID="577e3a93-0560-4099-bddb-dd55563d002d"
Jul 12 17:03:46.374009 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:46.373976    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gxtt8"
Jul 12 17:03:46.374427 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:03:46.374100    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gxtt8" podUID="259ec8ed-10bd-4fae-8f7f-97409bd4f54b"
Jul 12 17:03:46.628150 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:46.628081    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ltcld" event={"ID":"59055e87-8c03-4225-97e1-62114754af67","Type":"ContainerStarted","Data":"a283834731232622cc186478fcbdc65e3c579cbdba49300fd4e6ea9a7ad2965b"}
Jul 12 17:03:47.373389 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:47.373361    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h442x"
Jul 12 17:03:47.373536 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:03:47.373455    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h442x" podUID="f1d6c71b-d9fb-4f2a-9a81-5e8f0f8a6c13"
Jul 12 17:03:47.373536 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:47.373526    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mkw2b"
Jul 12 17:03:47.373634 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:03:47.373615    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mkw2b" podUID="577e3a93-0560-4099-bddb-dd55563d002d"
Jul 12 17:03:47.632813 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:47.632730    2569 generic.go:356] "Generic (PLEG): container finished" podID="59055e87-8c03-4225-97e1-62114754af67" containerID="a283834731232622cc186478fcbdc65e3c579cbdba49300fd4e6ea9a7ad2965b" exitCode=0
Jul 12 17:03:47.632813 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:47.632793    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ltcld" event={"ID":"59055e87-8c03-4225-97e1-62114754af67","Type":"ContainerDied","Data":"a283834731232622cc186478fcbdc65e3c579cbdba49300fd4e6ea9a7ad2965b"}
Jul 12 17:03:48.376928 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:48.376900    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gxtt8"
Jul 12 17:03:48.377079 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:03:48.377037    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gxtt8" podUID="259ec8ed-10bd-4fae-8f7f-97409bd4f54b"
Jul 12 17:03:48.638087 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:48.638007    2569 generic.go:356] "Generic (PLEG): container finished" podID="59055e87-8c03-4225-97e1-62114754af67" containerID="83b6cc6badf840c6d72fb4f5eeefcc7c383af7703f2fcda22956a272a9ba1fd2" exitCode=0
Jul 12 17:03:48.638087 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:48.638072    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ltcld" event={"ID":"59055e87-8c03-4225-97e1-62114754af67","Type":"ContainerDied","Data":"83b6cc6badf840c6d72fb4f5eeefcc7c383af7703f2fcda22956a272a9ba1fd2"}
Jul 12 17:03:49.374131 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:49.374097    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h442x"
Jul 12 17:03:49.374298 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:03:49.374202    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h442x" podUID="f1d6c71b-d9fb-4f2a-9a81-5e8f0f8a6c13"
Jul 12 17:03:49.374298 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:49.374259    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mkw2b"
Jul 12 17:03:49.374370 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:03:49.374338    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mkw2b" podUID="577e3a93-0560-4099-bddb-dd55563d002d"
Jul 12 17:03:49.643171 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:49.643094    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ltcld" event={"ID":"59055e87-8c03-4225-97e1-62114754af67","Type":"ContainerStarted","Data":"fb8d36db4155d4f553735e64094d5bef3bce06692698c9e60e276a28e7be1a7f"}
Jul 12 17:03:49.668640 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:49.668592    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-ltcld" podStartSLOduration=8.109457888 podStartE2EDuration="37.668577686s" totalImagesPullingTime="29.559119798s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-12 17:03:12 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 17:03:49.667206298 +0000 UTC m=+37.787523094" watchObservedRunningTime="2026-07-12 17:03:49.668577686 +0000 UTC m=+37.788894483"
Jul 12 17:03:50.373947 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:50.373878    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gxtt8"
Jul 12 17:03:50.374117 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:03:50.374027    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gxtt8" podUID="259ec8ed-10bd-4fae-8f7f-97409bd4f54b"
Jul 12 17:03:51.374270 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:51.374237    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h442x"
Jul 12 17:03:51.374859 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:51.374244    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mkw2b"
Jul 12 17:03:51.374859 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:03:51.374350    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h442x" podUID="f1d6c71b-d9fb-4f2a-9a81-5e8f0f8a6c13"
Jul 12 17:03:51.374859 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:03:51.374418    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mkw2b" podUID="577e3a93-0560-4099-bddb-dd55563d002d"
Jul 12 17:03:52.374273 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:52.374239    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gxtt8"
Jul 12 17:03:52.374731 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:03:52.374321    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gxtt8" podUID="259ec8ed-10bd-4fae-8f7f-97409bd4f54b"
Jul 12 17:03:52.934192 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:52.933939    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-mkw2b"]
Jul 12 17:03:52.934359 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:52.934282    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mkw2b"
Jul 12 17:03:52.934440 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:03:52.934415    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mkw2b" podUID="577e3a93-0560-4099-bddb-dd55563d002d"
Jul 12 17:03:52.934823 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:52.934627    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-gxtt8"]
Jul 12 17:03:52.934823 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:52.934704    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gxtt8"
Jul 12 17:03:52.934823 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:03:52.934786    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gxtt8" podUID="259ec8ed-10bd-4fae-8f7f-97409bd4f54b"
Jul 12 17:03:52.937116 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:52.937093    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-h442x"]
Jul 12 17:03:52.937234 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:52.937178    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h442x"
Jul 12 17:03:52.937308 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:03:52.937287    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h442x" podUID="f1d6c71b-d9fb-4f2a-9a81-5e8f0f8a6c13"
Jul 12 17:03:54.373557 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:54.373520    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mkw2b"
Jul 12 17:03:54.373557 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:54.373535    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gxtt8"
Jul 12 17:03:54.373557 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:54.373553    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h442x"
Jul 12 17:03:54.373999 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:03:54.373635    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mkw2b" podUID="577e3a93-0560-4099-bddb-dd55563d002d"
Jul 12 17:03:54.373999 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:03:54.373683    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gxtt8" podUID="259ec8ed-10bd-4fae-8f7f-97409bd4f54b"
Jul 12 17:03:54.373999 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:03:54.373783    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h442x" podUID="f1d6c71b-d9fb-4f2a-9a81-5e8f0f8a6c13"
Jul 12 17:03:56.373241 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:56.373213    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h442x"
Jul 12 17:03:56.373241 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:56.373240    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gxtt8"
Jul 12 17:03:56.373668 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:56.373222    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mkw2b"
Jul 12 17:03:56.373668 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:03:56.373324    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h442x" podUID="f1d6c71b-d9fb-4f2a-9a81-5e8f0f8a6c13"
Jul 12 17:03:56.373668 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:03:56.373393    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mkw2b" podUID="577e3a93-0560-4099-bddb-dd55563d002d"
Jul 12 17:03:56.373668 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:03:56.373438    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gxtt8" podUID="259ec8ed-10bd-4fae-8f7f-97409bd4f54b"
Jul 12 17:03:57.165272 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:57.165248    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-94.ec2.internal" event="NodeReady"
Jul 12 17:03:57.165395 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:57.165337    2569 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 12 17:03:57.210475 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:57.210453    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-2hvcl"]
Jul 12 17:03:57.242117 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:57.242058    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-ttcq7"]
Jul 12 17:03:57.242207 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:57.242176    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-2hvcl"
Jul 12 17:03:57.246240 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:57.246190    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 12 17:03:57.246240 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:57.246190    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 12 17:03:57.246408 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:57.246325    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qvzd7\""
Jul 12 17:03:57.274750 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:57.274727    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-2hvcl"]
Jul 12 17:03:57.274842 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:57.274758    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-ttcq7"]
Jul 12 17:03:57.274842 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:57.274773    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-xztsd"]
Jul 12 17:03:57.274842 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:57.274832    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-ttcq7"
Jul 12 17:03:57.277087 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:57.277062    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 12 17:03:57.277087 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:57.277079    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-xwq56\""
Jul 12 17:03:57.277241 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:57.277085    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 12 17:03:57.277241 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:57.277113    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 12 17:03:57.298680 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:57.298662    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-xztsd"]
Jul 12 17:03:57.298759 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:57.298750    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-xztsd"
Jul 12 17:03:57.301063 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:57.301037    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 12 17:03:57.301166 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:57.301081    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 12 17:03:57.301166 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:57.301099    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-vs9jl\""
Jul 12 17:03:57.301300 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:57.301288    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 12 17:03:57.301404 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:57.301389    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 12 17:03:57.380801 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:57.380768    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/09c551dc-74e0-4241-9d84-2cd264d717d9-cert\") pod \"ingress-canary-ttcq7\" (UID: \"09c551dc-74e0-4241-9d84-2cd264d717d9\") " pod="openshift-ingress-canary/ingress-canary-ttcq7"
Jul 12 17:03:57.381132 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:57.380808    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqcfs\" (UniqueName: \"kubernetes.io/projected/e97cfac9-53f7-40e4-97ed-fb725d04971c-kube-api-access-qqcfs\") pod \"dns-default-2hvcl\" (UID: \"e97cfac9-53f7-40e4-97ed-fb725d04971c\") " pod="openshift-dns/dns-default-2hvcl"
Jul 12 17:03:57.381132 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:57.380899    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e97cfac9-53f7-40e4-97ed-fb725d04971c-tmp-dir\") pod \"dns-default-2hvcl\" (UID: \"e97cfac9-53f7-40e4-97ed-fb725d04971c\") " pod="openshift-dns/dns-default-2hvcl"
Jul 12 17:03:57.381132 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:57.380934    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e97cfac9-53f7-40e4-97ed-fb725d04971c-metrics-tls\") pod \"dns-default-2hvcl\" (UID: \"e97cfac9-53f7-40e4-97ed-fb725d04971c\") " pod="openshift-dns/dns-default-2hvcl"
Jul 12 17:03:57.381132 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:57.380962    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e97cfac9-53f7-40e4-97ed-fb725d04971c-config-volume\") pod \"dns-default-2hvcl\" (UID: \"e97cfac9-53f7-40e4-97ed-fb725d04971c\") " pod="openshift-dns/dns-default-2hvcl"
Jul 12 17:03:57.381132 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:57.380980    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmfk8\" (UniqueName: \"kubernetes.io/projected/09c551dc-74e0-4241-9d84-2cd264d717d9-kube-api-access-cmfk8\") pod \"ingress-canary-ttcq7\" (UID: \"09c551dc-74e0-4241-9d84-2cd264d717d9\") " pod="openshift-ingress-canary/ingress-canary-ttcq7"
Jul 12 17:03:57.482014 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:57.481985    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e97cfac9-53f7-40e4-97ed-fb725d04971c-config-volume\") pod \"dns-default-2hvcl\" (UID: \"e97cfac9-53f7-40e4-97ed-fb725d04971c\") " pod="openshift-dns/dns-default-2hvcl"
Jul 12 17:03:57.482014 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:57.482016    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmfk8\" (UniqueName: \"kubernetes.io/projected/09c551dc-74e0-4241-9d84-2cd264d717d9-kube-api-access-cmfk8\") pod \"ingress-canary-ttcq7\" (UID: \"09c551dc-74e0-4241-9d84-2cd264d717d9\") " pod="openshift-ingress-canary/ingress-canary-ttcq7"
Jul 12 17:03:57.482178 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:57.482035    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5c7f573f-607f-40cd-b450-1976b2898b83-data-volume\") pod \"insights-runtime-extractor-xztsd\" (UID: \"5c7f573f-607f-40cd-b450-1976b2898b83\") " pod="openshift-insights/insights-runtime-extractor-xztsd"
Jul 12 17:03:57.482178 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:57.482057    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5c7f573f-607f-40cd-b450-1976b2898b83-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xztsd\" (UID: \"5c7f573f-607f-40cd-b450-1976b2898b83\") " pod="openshift-insights/insights-runtime-extractor-xztsd"
Jul 12 17:03:57.482178 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:57.482121    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/09c551dc-74e0-4241-9d84-2cd264d717d9-cert\") pod \"ingress-canary-ttcq7\" (UID: \"09c551dc-74e0-4241-9d84-2cd264d717d9\") " pod="openshift-ingress-canary/ingress-canary-ttcq7"
Jul 12 17:03:57.482178 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:57.482148    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqcfs\" (UniqueName: \"kubernetes.io/projected/e97cfac9-53f7-40e4-97ed-fb725d04971c-kube-api-access-qqcfs\") pod \"dns-default-2hvcl\" (UID: \"e97cfac9-53f7-40e4-97ed-fb725d04971c\") " pod="openshift-dns/dns-default-2hvcl"
Jul 12 17:03:57.482379 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:57.482187    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5c7f573f-607f-40cd-b450-1976b2898b83-crio-socket\") pod \"insights-runtime-extractor-xztsd\" (UID: \"5c7f573f-607f-40cd-b450-1976b2898b83\") " pod="openshift-insights/insights-runtime-extractor-xztsd"
Jul 12 17:03:57.482379 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:57.482209    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5c7f573f-607f-40cd-b450-1976b2898b83-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xztsd\" (UID: \"5c7f573f-607f-40cd-b450-1976b2898b83\") " pod="openshift-insights/insights-runtime-extractor-xztsd"
Jul 12 17:03:57.482379 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:57.482245    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e97cfac9-53f7-40e4-97ed-fb725d04971c-tmp-dir\") pod \"dns-default-2hvcl\" (UID: \"e97cfac9-53f7-40e4-97ed-fb725d04971c\") " pod="openshift-dns/dns-default-2hvcl"
Jul 12 17:03:57.482379 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:57.482292    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e97cfac9-53f7-40e4-97ed-fb725d04971c-metrics-tls\") pod \"dns-default-2hvcl\" (UID: \"e97cfac9-53f7-40e4-97ed-fb725d04971c\") " pod="openshift-dns/dns-default-2hvcl"
Jul 12 17:03:57.482379 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:57.482345    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqhrs\" (UniqueName: \"kubernetes.io/projected/5c7f573f-607f-40cd-b450-1976b2898b83-kube-api-access-fqhrs\") pod \"insights-runtime-extractor-xztsd\" (UID: \"5c7f573f-607f-40cd-b450-1976b2898b83\") " pod="openshift-insights/insights-runtime-extractor-xztsd"
Jul 12 17:03:57.482662 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:57.482475    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e97cfac9-53f7-40e4-97ed-fb725d04971c-tmp-dir\") pod \"dns-default-2hvcl\" (UID: \"e97cfac9-53f7-40e4-97ed-fb725d04971c\") " pod="openshift-dns/dns-default-2hvcl"
Jul 12 17:03:57.482662 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:57.482639    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e97cfac9-53f7-40e4-97ed-fb725d04971c-config-volume\") pod \"dns-default-2hvcl\" (UID: \"e97cfac9-53f7-40e4-97ed-fb725d04971c\") " pod="openshift-dns/dns-default-2hvcl"
Jul 12 17:03:57.486114 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:57.486089    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e97cfac9-53f7-40e4-97ed-fb725d04971c-metrics-tls\") pod \"dns-default-2hvcl\" (UID: \"e97cfac9-53f7-40e4-97ed-fb725d04971c\") " pod="openshift-dns/dns-default-2hvcl"
Jul 12 17:03:57.486197 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:57.486138    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/09c551dc-74e0-4241-9d84-2cd264d717d9-cert\") pod \"ingress-canary-ttcq7\" (UID: \"09c551dc-74e0-4241-9d84-2cd264d717d9\") " pod="openshift-ingress-canary/ingress-canary-ttcq7"
Jul 12 17:03:57.489993 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:57.489967    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqcfs\" (UniqueName: \"kubernetes.io/projected/e97cfac9-53f7-40e4-97ed-fb725d04971c-kube-api-access-qqcfs\") pod \"dns-default-2hvcl\" (UID: \"e97cfac9-53f7-40e4-97ed-fb725d04971c\") " pod="openshift-dns/dns-default-2hvcl"
Jul 12 17:03:57.490190 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:57.490158    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmfk8\" (UniqueName: \"kubernetes.io/projected/09c551dc-74e0-4241-9d84-2cd264d717d9-kube-api-access-cmfk8\") pod \"ingress-canary-ttcq7\" (UID: \"09c551dc-74e0-4241-9d84-2cd264d717d9\") " pod="openshift-ingress-canary/ingress-canary-ttcq7"
Jul 12 17:03:57.555115 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:57.555040    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-2hvcl"
Jul 12 17:03:57.583181 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:57.583159    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-ttcq7"
Jul 12 17:03:57.583320 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:57.583302    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqhrs\" (UniqueName: \"kubernetes.io/projected/5c7f573f-607f-40cd-b450-1976b2898b83-kube-api-access-fqhrs\") pod \"insights-runtime-extractor-xztsd\" (UID: \"5c7f573f-607f-40cd-b450-1976b2898b83\") " pod="openshift-insights/insights-runtime-extractor-xztsd"
Jul 12 17:03:57.583370 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:57.583334    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5c7f573f-607f-40cd-b450-1976b2898b83-data-volume\") pod \"insights-runtime-extractor-xztsd\" (UID: \"5c7f573f-607f-40cd-b450-1976b2898b83\") " pod="openshift-insights/insights-runtime-extractor-xztsd"
Jul 12 17:03:57.583370 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:57.583360    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5c7f573f-607f-40cd-b450-1976b2898b83-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xztsd\" (UID: \"5c7f573f-607f-40cd-b450-1976b2898b83\") " pod="openshift-insights/insights-runtime-extractor-xztsd"
Jul 12 17:03:57.583547 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:57.583525    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5c7f573f-607f-40cd-b450-1976b2898b83-crio-socket\") pod \"insights-runtime-extractor-xztsd\" (UID: \"5c7f573f-607f-40cd-b450-1976b2898b83\") " pod="openshift-insights/insights-runtime-extractor-xztsd"
Jul 12 17:03:57.583628 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:57.583564    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5c7f573f-607f-40cd-b450-1976b2898b83-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xztsd\" (UID: \"5c7f573f-607f-40cd-b450-1976b2898b83\") " pod="openshift-insights/insights-runtime-extractor-xztsd"
Jul 12 17:03:57.583703 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:57.583685    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5c7f573f-607f-40cd-b450-1976b2898b83-data-volume\") pod \"insights-runtime-extractor-xztsd\" (UID: \"5c7f573f-607f-40cd-b450-1976b2898b83\") " pod="openshift-insights/insights-runtime-extractor-xztsd"
Jul 12 17:03:57.583757 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:57.583738    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5c7f573f-607f-40cd-b450-1976b2898b83-crio-socket\") pod \"insights-runtime-extractor-xztsd\" (UID: \"5c7f573f-607f-40cd-b450-1976b2898b83\") " pod="openshift-insights/insights-runtime-extractor-xztsd"
Jul 12 17:03:57.583913 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:57.583886    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5c7f573f-607f-40cd-b450-1976b2898b83-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xztsd\" (UID: \"5c7f573f-607f-40cd-b450-1976b2898b83\") " pod="openshift-insights/insights-runtime-extractor-xztsd"
Jul 12 17:03:57.585587 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:57.585567    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5c7f573f-607f-40cd-b450-1976b2898b83-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xztsd\" (UID: \"5c7f573f-607f-40cd-b450-1976b2898b83\") " pod="openshift-insights/insights-runtime-extractor-xztsd"
Jul 12 17:03:57.591941 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:57.591925    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqhrs\" (UniqueName: \"kubernetes.io/projected/5c7f573f-607f-40cd-b450-1976b2898b83-kube-api-access-fqhrs\") pod \"insights-runtime-extractor-xztsd\" (UID: \"5c7f573f-607f-40cd-b450-1976b2898b83\") " pod="openshift-insights/insights-runtime-extractor-xztsd"
Jul 12 17:03:57.607002 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:57.606984    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-xztsd"
Jul 12 17:03:57.765559 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:57.765532    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-ttcq7"]
Jul 12 17:03:57.768856 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:57.768831    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09c551dc_74e0_4241_9d84_2cd264d717d9.slice/crio-27071c694a5bab3fbd4a30538d69f89eeb6dd98e0dddd5a0d4df43d6f14b0669 WatchSource:0}: Error finding container 27071c694a5bab3fbd4a30538d69f89eeb6dd98e0dddd5a0d4df43d6f14b0669: Status 404 returned error can't find the container with id 27071c694a5bab3fbd4a30538d69f89eeb6dd98e0dddd5a0d4df43d6f14b0669
Jul 12 17:03:57.771300 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:57.771202    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-2hvcl"]
Jul 12 17:03:57.775597 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:57.775570    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode97cfac9_53f7_40e4_97ed_fb725d04971c.slice/crio-e41d1d7559ad145bffafdbd3a3a8c8bc0ad7f5b02544c825e2bd495748d9c2c5 WatchSource:0}: Error finding container e41d1d7559ad145bffafdbd3a3a8c8bc0ad7f5b02544c825e2bd495748d9c2c5: Status 404 returned error can't find the container with id e41d1d7559ad145bffafdbd3a3a8c8bc0ad7f5b02544c825e2bd495748d9c2c5
Jul 12 17:03:57.783027 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:57.783004    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-xztsd"]
Jul 12 17:03:57.786074 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:57.786054    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c7f573f_607f_40cd_b450_1976b2898b83.slice/crio-a30100ac337c5ef92d988088bdf12d3902e0eb8f76d69706ab15c6a7d887ad9f WatchSource:0}: Error finding container a30100ac337c5ef92d988088bdf12d3902e0eb8f76d69706ab15c6a7d887ad9f: Status 404 returned error can't find the container with id a30100ac337c5ef92d988088bdf12d3902e0eb8f76d69706ab15c6a7d887ad9f
Jul 12 17:03:58.373777 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:58.373740    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mkw2b"
Jul 12 17:03:58.373976 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:58.373753    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h442x"
Jul 12 17:03:58.374376 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:58.373754    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gxtt8"
Jul 12 17:03:58.377572 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:58.377377    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 12 17:03:58.377572 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:58.377415    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-fgjd4\""
Jul 12 17:03:58.377572 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:58.377433    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 12 17:03:58.377572 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:58.377442    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 12 17:03:58.377572 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:58.377498    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 12 17:03:58.377878 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:58.377764    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-wpssr\""
Jul 12 17:03:58.510400 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:58.509426    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-jkqx9"]
Jul 12 17:03:58.523225 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:58.522315    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-64785b6675-jkqx9"
Jul 12 17:03:58.527037 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:58.526329    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 12 17:03:58.527037 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:58.526377    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-bgwsg\""
Jul 12 17:03:58.527037 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:58.527032    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 12 17:03:58.527325 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:58.527253    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 12 17:03:58.527325 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:58.527273    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 12 17:03:58.527877 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:58.527480    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 12 17:03:58.528454 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:58.528397    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-jkqx9"]
Jul 12 17:03:58.666232 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:58.666078    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xztsd" event={"ID":"5c7f573f-607f-40cd-b450-1976b2898b83","Type":"ContainerStarted","Data":"a30100ac337c5ef92d988088bdf12d3902e0eb8f76d69706ab15c6a7d887ad9f"}
Jul 12 17:03:58.668151 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:58.668047    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-ttcq7" event={"ID":"09c551dc-74e0-4241-9d84-2cd264d717d9","Type":"ContainerStarted","Data":"27071c694a5bab3fbd4a30538d69f89eeb6dd98e0dddd5a0d4df43d6f14b0669"}
Jul 12 17:03:58.669497 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:58.669454    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2hvcl" event={"ID":"e97cfac9-53f7-40e4-97ed-fb725d04971c","Type":"ContainerStarted","Data":"e41d1d7559ad145bffafdbd3a3a8c8bc0ad7f5b02544c825e2bd495748d9c2c5"}
Jul 12 17:03:58.690378 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:58.690348    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/23b6cfce-b79a-4c63-9c76-1932981d702b-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-jkqx9\" (UID: \"23b6cfce-b79a-4c63-9c76-1932981d702b\") " pod="openshift-monitoring/prometheus-operator-64785b6675-jkqx9"
Jul 12 17:03:58.690480 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:58.690422    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/23b6cfce-b79a-4c63-9c76-1932981d702b-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-jkqx9\" (UID: \"23b6cfce-b79a-4c63-9c76-1932981d702b\") " pod="openshift-monitoring/prometheus-operator-64785b6675-jkqx9"
Jul 12 17:03:58.690480 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:58.690454    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4dm5\" (UniqueName: \"kubernetes.io/projected/23b6cfce-b79a-4c63-9c76-1932981d702b-kube-api-access-d4dm5\") pod \"prometheus-operator-64785b6675-jkqx9\" (UID: \"23b6cfce-b79a-4c63-9c76-1932981d702b\") " pod="openshift-monitoring/prometheus-operator-64785b6675-jkqx9"
Jul 12 17:03:58.690610 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:58.690494    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/23b6cfce-b79a-4c63-9c76-1932981d702b-metrics-client-ca\") pod \"prometheus-operator-64785b6675-jkqx9\" (UID: \"23b6cfce-b79a-4c63-9c76-1932981d702b\") " pod="openshift-monitoring/prometheus-operator-64785b6675-jkqx9"
Jul 12 17:03:58.791693 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:58.791659    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4dm5\" (UniqueName: \"kubernetes.io/projected/23b6cfce-b79a-4c63-9c76-1932981d702b-kube-api-access-d4dm5\") pod \"prometheus-operator-64785b6675-jkqx9\" (UID: \"23b6cfce-b79a-4c63-9c76-1932981d702b\") " pod="openshift-monitoring/prometheus-operator-64785b6675-jkqx9"
Jul 12 17:03:58.791842 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:58.791791    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/23b6cfce-b79a-4c63-9c76-1932981d702b-metrics-client-ca\") pod \"prometheus-operator-64785b6675-jkqx9\" (UID: \"23b6cfce-b79a-4c63-9c76-1932981d702b\") " pod="openshift-monitoring/prometheus-operator-64785b6675-jkqx9"
Jul 12 17:03:58.791901 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:58.791870    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/23b6cfce-b79a-4c63-9c76-1932981d702b-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-jkqx9\" (UID: \"23b6cfce-b79a-4c63-9c76-1932981d702b\") " pod="openshift-monitoring/prometheus-operator-64785b6675-jkqx9"
Jul 12 17:03:58.791969 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:58.791938    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/23b6cfce-b79a-4c63-9c76-1932981d702b-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-jkqx9\" (UID: \"23b6cfce-b79a-4c63-9c76-1932981d702b\") " pod="openshift-monitoring/prometheus-operator-64785b6675-jkqx9"
Jul 12 17:03:58.792079 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:03:58.792064    2569 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Jul 12 17:03:58.792139 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:03:58.792129    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/23b6cfce-b79a-4c63-9c76-1932981d702b-prometheus-operator-tls podName:23b6cfce-b79a-4c63-9c76-1932981d702b nodeName:}" failed. No retries permitted until 2026-07-12 17:03:59.292110275 +0000 UTC m=+47.412427054 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/23b6cfce-b79a-4c63-9c76-1932981d702b-prometheus-operator-tls") pod "prometheus-operator-64785b6675-jkqx9" (UID: "23b6cfce-b79a-4c63-9c76-1932981d702b") : secret "prometheus-operator-tls" not found
Jul 12 17:03:58.793099 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:58.793072    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/23b6cfce-b79a-4c63-9c76-1932981d702b-metrics-client-ca\") pod \"prometheus-operator-64785b6675-jkqx9\" (UID: \"23b6cfce-b79a-4c63-9c76-1932981d702b\") " pod="openshift-monitoring/prometheus-operator-64785b6675-jkqx9"
Jul 12 17:03:58.798489 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:58.798462    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/23b6cfce-b79a-4c63-9c76-1932981d702b-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-jkqx9\" (UID: \"23b6cfce-b79a-4c63-9c76-1932981d702b\") " pod="openshift-monitoring/prometheus-operator-64785b6675-jkqx9"
Jul 12 17:03:58.813619 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:58.813490    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4dm5\" (UniqueName: \"kubernetes.io/projected/23b6cfce-b79a-4c63-9c76-1932981d702b-kube-api-access-d4dm5\") pod \"prometheus-operator-64785b6675-jkqx9\" (UID: \"23b6cfce-b79a-4c63-9c76-1932981d702b\") " pod="openshift-monitoring/prometheus-operator-64785b6675-jkqx9"
Jul 12 17:03:59.074812 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:59.074782    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-55b6b74c7-5jbs4"]
Jul 12 17:03:59.098660 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:59.098629    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-55b6b74c7-5jbs4"]
Jul 12 17:03:59.098812 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:59.098750    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55b6b74c7-5jbs4"
Jul 12 17:03:59.101599 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:59.101544    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 12 17:03:59.101751 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:59.101738    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-2nhhq\""
Jul 12 17:03:59.101822 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:59.101772    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 12 17:03:59.101822 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:59.101793    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 12 17:03:59.101967 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:59.101886    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 12 17:03:59.102018 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:59.101976    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 12 17:03:59.102391 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:59.102358    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 12 17:03:59.102520 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:59.102442    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 12 17:03:59.194294 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:59.194262    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/37783a93-d722-4cb5-b5d3-7e1cb8726990-oauth-serving-cert\") pod \"console-55b6b74c7-5jbs4\" (UID: \"37783a93-d722-4cb5-b5d3-7e1cb8726990\") " pod="openshift-console/console-55b6b74c7-5jbs4"
Jul 12 17:03:59.194451 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:59.194322    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/37783a93-d722-4cb5-b5d3-7e1cb8726990-console-oauth-config\") pod \"console-55b6b74c7-5jbs4\" (UID: \"37783a93-d722-4cb5-b5d3-7e1cb8726990\") " pod="openshift-console/console-55b6b74c7-5jbs4"
Jul 12 17:03:59.194451 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:59.194349    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vw8w6\" (UniqueName: \"kubernetes.io/projected/37783a93-d722-4cb5-b5d3-7e1cb8726990-kube-api-access-vw8w6\") pod \"console-55b6b74c7-5jbs4\" (UID: \"37783a93-d722-4cb5-b5d3-7e1cb8726990\") " pod="openshift-console/console-55b6b74c7-5jbs4"
Jul 12 17:03:59.194451 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:59.194412    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/37783a93-d722-4cb5-b5d3-7e1cb8726990-console-config\") pod \"console-55b6b74c7-5jbs4\" (UID: \"37783a93-d722-4cb5-b5d3-7e1cb8726990\") " pod="openshift-console/console-55b6b74c7-5jbs4"
Jul 12 17:03:59.194451 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:59.194441    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/37783a93-d722-4cb5-b5d3-7e1cb8726990-service-ca\") pod \"console-55b6b74c7-5jbs4\" (UID: \"37783a93-d722-4cb5-b5d3-7e1cb8726990\") " pod="openshift-console/console-55b6b74c7-5jbs4"
Jul 12 17:03:59.194677 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:59.194479    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/37783a93-d722-4cb5-b5d3-7e1cb8726990-console-serving-cert\") pod \"console-55b6b74c7-5jbs4\" (UID: \"37783a93-d722-4cb5-b5d3-7e1cb8726990\") " pod="openshift-console/console-55b6b74c7-5jbs4"
Jul 12 17:03:59.295822 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:59.295785    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/23b6cfce-b79a-4c63-9c76-1932981d702b-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-jkqx9\" (UID: \"23b6cfce-b79a-4c63-9c76-1932981d702b\") " pod="openshift-monitoring/prometheus-operator-64785b6675-jkqx9"
Jul 12 17:03:59.295998 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:59.295830    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/37783a93-d722-4cb5-b5d3-7e1cb8726990-console-config\") pod \"console-55b6b74c7-5jbs4\" (UID: \"37783a93-d722-4cb5-b5d3-7e1cb8726990\") " pod="openshift-console/console-55b6b74c7-5jbs4"
Jul 12 17:03:59.295998 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:59.295861    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/37783a93-d722-4cb5-b5d3-7e1cb8726990-service-ca\") pod \"console-55b6b74c7-5jbs4\" (UID: \"37783a93-d722-4cb5-b5d3-7e1cb8726990\") " pod="openshift-console/console-55b6b74c7-5jbs4"
Jul 12 17:03:59.295998 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:59.295899    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/37783a93-d722-4cb5-b5d3-7e1cb8726990-console-serving-cert\") pod \"console-55b6b74c7-5jbs4\" (UID: \"37783a93-d722-4cb5-b5d3-7e1cb8726990\") " pod="openshift-console/console-55b6b74c7-5jbs4"
Jul 12 17:03:59.295998 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:59.295954    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/37783a93-d722-4cb5-b5d3-7e1cb8726990-oauth-serving-cert\") pod \"console-55b6b74c7-5jbs4\" (UID: \"37783a93-d722-4cb5-b5d3-7e1cb8726990\") " pod="openshift-console/console-55b6b74c7-5jbs4"
Jul 12 17:03:59.295998 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:59.295994    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/37783a93-d722-4cb5-b5d3-7e1cb8726990-console-oauth-config\") pod \"console-55b6b74c7-5jbs4\" (UID: \"37783a93-d722-4cb5-b5d3-7e1cb8726990\") " pod="openshift-console/console-55b6b74c7-5jbs4"
Jul 12 17:03:59.296247 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:59.296020    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vw8w6\" (UniqueName: \"kubernetes.io/projected/37783a93-d722-4cb5-b5d3-7e1cb8726990-kube-api-access-vw8w6\") pod \"console-55b6b74c7-5jbs4\" (UID: \"37783a93-d722-4cb5-b5d3-7e1cb8726990\") " pod="openshift-console/console-55b6b74c7-5jbs4"
Jul 12 17:03:59.297120 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:59.297096    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/37783a93-d722-4cb5-b5d3-7e1cb8726990-service-ca\") pod \"console-55b6b74c7-5jbs4\" (UID: \"37783a93-d722-4cb5-b5d3-7e1cb8726990\") " pod="openshift-console/console-55b6b74c7-5jbs4"
Jul 12 17:03:59.297808 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:59.297764    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/37783a93-d722-4cb5-b5d3-7e1cb8726990-oauth-serving-cert\") pod \"console-55b6b74c7-5jbs4\" (UID: \"37783a93-d722-4cb5-b5d3-7e1cb8726990\") " pod="openshift-console/console-55b6b74c7-5jbs4"
Jul 12 17:03:59.297808 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:59.297779    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/37783a93-d722-4cb5-b5d3-7e1cb8726990-console-config\") pod \"console-55b6b74c7-5jbs4\" (UID: \"37783a93-d722-4cb5-b5d3-7e1cb8726990\") " pod="openshift-console/console-55b6b74c7-5jbs4"
Jul 12 17:03:59.300633 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:59.300608    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/37783a93-d722-4cb5-b5d3-7e1cb8726990-console-serving-cert\") pod \"console-55b6b74c7-5jbs4\" (UID: \"37783a93-d722-4cb5-b5d3-7e1cb8726990\") " pod="openshift-console/console-55b6b74c7-5jbs4"
Jul 12 17:03:59.300901 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:59.300841    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/23b6cfce-b79a-4c63-9c76-1932981d702b-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-jkqx9\" (UID: \"23b6cfce-b79a-4c63-9c76-1932981d702b\") " pod="openshift-monitoring/prometheus-operator-64785b6675-jkqx9"
Jul 12 17:03:59.302809 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:59.302770    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/37783a93-d722-4cb5-b5d3-7e1cb8726990-console-oauth-config\") pod \"console-55b6b74c7-5jbs4\" (UID: \"37783a93-d722-4cb5-b5d3-7e1cb8726990\") " pod="openshift-console/console-55b6b74c7-5jbs4"
Jul 12 17:03:59.305562 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:59.305533    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vw8w6\" (UniqueName: \"kubernetes.io/projected/37783a93-d722-4cb5-b5d3-7e1cb8726990-kube-api-access-vw8w6\") pod \"console-55b6b74c7-5jbs4\" (UID: \"37783a93-d722-4cb5-b5d3-7e1cb8726990\") " pod="openshift-console/console-55b6b74c7-5jbs4"
Jul 12 17:03:59.413770 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:59.413702    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55b6b74c7-5jbs4"
Jul 12 17:03:59.436710 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:59.436567    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-64785b6675-jkqx9"
Jul 12 17:03:59.617108 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:59.617035    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-55b6b74c7-5jbs4"]
Jul 12 17:03:59.649021 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:59.648945    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-jkqx9"]
Jul 12 17:03:59.653076 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:03:59.653048    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23b6cfce_b79a_4c63_9c76_1932981d702b.slice/crio-e61f3b6ee520dbc8fe53e9cc988b11508d1f07948ecd0d7932c6751de2cf1049 WatchSource:0}: Error finding container e61f3b6ee520dbc8fe53e9cc988b11508d1f07948ecd0d7932c6751de2cf1049: Status 404 returned error can't find the container with id e61f3b6ee520dbc8fe53e9cc988b11508d1f07948ecd0d7932c6751de2cf1049
Jul 12 17:03:59.675664 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:59.675633    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55b6b74c7-5jbs4" event={"ID":"37783a93-d722-4cb5-b5d3-7e1cb8726990","Type":"ContainerStarted","Data":"a460a27b17a0480a5c06215b6aeae1080bc6e6aee0bda0fbfde0d2b63c8873be"}
Jul 12 17:03:59.677381 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:03:59.677351    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-jkqx9" event={"ID":"23b6cfce-b79a-4c63-9c76-1932981d702b","Type":"ContainerStarted","Data":"e61f3b6ee520dbc8fe53e9cc988b11508d1f07948ecd0d7932c6751de2cf1049"}
Jul 12 17:04:06.696957 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:06.696922    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-ttcq7" event={"ID":"09c551dc-74e0-4241-9d84-2cd264d717d9","Type":"ContainerStarted","Data":"58c80b74b4b63a3fe70f4742fdd16b7b4c0d43c6822291e0c138426c91b8741d"}
Jul 12 17:04:06.698705 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:06.698675    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2hvcl" event={"ID":"e97cfac9-53f7-40e4-97ed-fb725d04971c","Type":"ContainerStarted","Data":"d556c87fa251dbe8d4e0ff9cac0c86865b597940bbbc96635d036b86374014ec"}
Jul 12 17:04:06.698826 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:06.698723    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2hvcl" event={"ID":"e97cfac9-53f7-40e4-97ed-fb725d04971c","Type":"ContainerStarted","Data":"8a0f9535d19d2302f1b673cc43aed10c7129d9a088d1a4d26772519e21463326"}
Jul 12 17:04:06.698826 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:06.698753    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-2hvcl"
Jul 12 17:04:06.700219 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:06.700191    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55b6b74c7-5jbs4" event={"ID":"37783a93-d722-4cb5-b5d3-7e1cb8726990","Type":"ContainerStarted","Data":"2678a40659b3c25aa857a7f3677eb25a9573bab7367a5e1704d3209c3c4f147d"}
Jul 12 17:04:06.701485 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:06.701457    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xztsd" event={"ID":"5c7f573f-607f-40cd-b450-1976b2898b83","Type":"ContainerStarted","Data":"89e671a78f6df1dd00787804f0f20ada62c4549106b22fd1898c695de0f7c809"}
Jul 12 17:04:06.703020 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:06.702997    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-jkqx9" event={"ID":"23b6cfce-b79a-4c63-9c76-1932981d702b","Type":"ContainerStarted","Data":"66789fe063d170cce67607fdff7c77b23bb6d767a7442b5160dc36e16dd90f5c"}
Jul 12 17:04:06.703086 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:06.703041    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-jkqx9" event={"ID":"23b6cfce-b79a-4c63-9c76-1932981d702b","Type":"ContainerStarted","Data":"e411c72bf8273fa161d03c34e5b3ccf80cff1f1bfb34889a734cf7b84dc247e7"}
Jul 12 17:04:06.712253 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:06.712211    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-ttcq7" podStartSLOduration=3.352690048 podStartE2EDuration="9.712200015s" totalImagesPullingTime="6.359509967s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 17:03:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 17:04:06.711574081 +0000 UTC m=+54.831890878" watchObservedRunningTime="2026-07-12 17:04:06.712200015 +0000 UTC m=+54.832516811"
Jul 12 17:04:06.732473 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:06.730442    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-2hvcl" podStartSLOduration=1.52355192 podStartE2EDuration="9.730424729s" totalImagesPullingTime="8.206872809s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 17:03:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 17:04:06.728352186 +0000 UTC m=+54.848668985" watchObservedRunningTime="2026-07-12 17:04:06.730424729 +0000 UTC m=+54.850741530"
Jul 12 17:04:06.747232 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:06.747193    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-55b6b74c7-5jbs4" podStartSLOduration=1.355465272 podStartE2EDuration="7.747180996s" totalImagesPullingTime="6.391715724s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 17:03:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 17:04:06.74638083 +0000 UTC m=+54.866697628" watchObservedRunningTime="2026-07-12 17:04:06.747180996 +0000 UTC m=+54.867497771"
Jul 12 17:04:06.766288 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:06.766115    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-64785b6675-jkqx9" podStartSLOduration=2.407026311 podStartE2EDuration="8.766103831s" totalImagesPullingTime="6.35907752s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 17:03:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 17:04:06.765439249 +0000 UTC m=+54.885756058" watchObservedRunningTime="2026-07-12 17:04:06.766103831 +0000 UTC m=+54.886420629"
Jul 12 17:04:07.708212 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:07.708172    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xztsd" event={"ID":"5c7f573f-607f-40cd-b450-1976b2898b83","Type":"ContainerStarted","Data":"75c321bdd210b5de36a258f877bf4b2791ce5c9c471faa9938321195f10d132f"}
Jul 12 17:04:08.893575 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:08.893543    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-x46rv"]
Jul 12 17:04:08.902365 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:08.902338    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-x46rv"
Jul 12 17:04:08.905003 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:08.904976    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Jul 12 17:04:08.905115 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:08.905004    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-9h2gj\""
Jul 12 17:04:08.905230 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:08.905216    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Jul 12 17:04:08.913026 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:08.913003    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-gc7xw"]
Jul 12 17:04:08.916246 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:08.916227    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-x46rv"]
Jul 12 17:04:08.916341 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:08.916332    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-gc7xw"
Jul 12 17:04:08.918737 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:08.918714    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 12 17:04:08.918878 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:08.918858    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 12 17:04:08.918968 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:08.918925    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-rbqq8\""
Jul 12 17:04:08.919636 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:08.919611    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 12 17:04:08.978240 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:08.978200    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/156026b1-c28b-43ca-9249-6c85d5de032e-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-x46rv\" (UID: \"156026b1-c28b-43ca-9249-6c85d5de032e\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-x46rv"
Jul 12 17:04:08.978397 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:08.978263    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/943fb073-b5af-467c-a4fe-16d9a0cb07bc-node-exporter-tls\") pod \"node-exporter-gc7xw\" (UID: \"943fb073-b5af-467c-a4fe-16d9a0cb07bc\") " pod="openshift-monitoring/node-exporter-gc7xw"
Jul 12 17:04:08.978397 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:08.978312    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/943fb073-b5af-467c-a4fe-16d9a0cb07bc-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gc7xw\" (UID: \"943fb073-b5af-467c-a4fe-16d9a0cb07bc\") " pod="openshift-monitoring/node-exporter-gc7xw"
Jul 12 17:04:08.978397 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:08.978357    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/943fb073-b5af-467c-a4fe-16d9a0cb07bc-node-exporter-accelerators-collector-config\") pod \"node-exporter-gc7xw\" (UID: \"943fb073-b5af-467c-a4fe-16d9a0cb07bc\") " pod="openshift-monitoring/node-exporter-gc7xw"
Jul 12 17:04:08.978569 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:08.978398    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/943fb073-b5af-467c-a4fe-16d9a0cb07bc-node-exporter-textfile\") pod \"node-exporter-gc7xw\" (UID: \"943fb073-b5af-467c-a4fe-16d9a0cb07bc\") " pod="openshift-monitoring/node-exporter-gc7xw"
Jul 12 17:04:08.978569 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:08.978466    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/943fb073-b5af-467c-a4fe-16d9a0cb07bc-node-exporter-wtmp\") pod \"node-exporter-gc7xw\" (UID: \"943fb073-b5af-467c-a4fe-16d9a0cb07bc\") " pod="openshift-monitoring/node-exporter-gc7xw"
Jul 12 17:04:08.978569 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:08.978540    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/156026b1-c28b-43ca-9249-6c85d5de032e-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-x46rv\" (UID: \"156026b1-c28b-43ca-9249-6c85d5de032e\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-x46rv"
Jul 12 17:04:08.978686 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:08.978594    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjj8k\" (UniqueName: \"kubernetes.io/projected/156026b1-c28b-43ca-9249-6c85d5de032e-kube-api-access-gjj8k\") pod \"openshift-state-metrics-84bd68b896-x46rv\" (UID: \"156026b1-c28b-43ca-9249-6c85d5de032e\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-x46rv"
Jul 12 17:04:08.978686 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:08.978622    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/943fb073-b5af-467c-a4fe-16d9a0cb07bc-root\") pod \"node-exporter-gc7xw\" (UID: \"943fb073-b5af-467c-a4fe-16d9a0cb07bc\") " pod="openshift-monitoring/node-exporter-gc7xw"
Jul 12 17:04:08.978686 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:08.978647    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/943fb073-b5af-467c-a4fe-16d9a0cb07bc-sys\") pod \"node-exporter-gc7xw\" (UID: \"943fb073-b5af-467c-a4fe-16d9a0cb07bc\") " pod="openshift-monitoring/node-exporter-gc7xw"
Jul 12 17:04:08.978802 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:08.978684    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/943fb073-b5af-467c-a4fe-16d9a0cb07bc-metrics-client-ca\") pod \"node-exporter-gc7xw\" (UID: \"943fb073-b5af-467c-a4fe-16d9a0cb07bc\") " pod="openshift-monitoring/node-exporter-gc7xw"
Jul 12 17:04:08.978802 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:08.978714    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sltq9\" (UniqueName: \"kubernetes.io/projected/943fb073-b5af-467c-a4fe-16d9a0cb07bc-kube-api-access-sltq9\") pod \"node-exporter-gc7xw\" (UID: \"943fb073-b5af-467c-a4fe-16d9a0cb07bc\") " pod="openshift-monitoring/node-exporter-gc7xw"
Jul 12 17:04:08.978802 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:08.978746    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/156026b1-c28b-43ca-9249-6c85d5de032e-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-x46rv\" (UID: \"156026b1-c28b-43ca-9249-6c85d5de032e\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-x46rv"
Jul 12 17:04:09.079526 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:09.079483    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/156026b1-c28b-43ca-9249-6c85d5de032e-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-x46rv\" (UID: \"156026b1-c28b-43ca-9249-6c85d5de032e\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-x46rv"
Jul 12 17:04:09.079727 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:09.079544    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/943fb073-b5af-467c-a4fe-16d9a0cb07bc-node-exporter-tls\") pod \"node-exporter-gc7xw\" (UID: \"943fb073-b5af-467c-a4fe-16d9a0cb07bc\") " pod="openshift-monitoring/node-exporter-gc7xw"
Jul 12 17:04:09.079727 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:09.079563    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/943fb073-b5af-467c-a4fe-16d9a0cb07bc-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gc7xw\" (UID: \"943fb073-b5af-467c-a4fe-16d9a0cb07bc\") " pod="openshift-monitoring/node-exporter-gc7xw"
Jul 12 17:04:09.079727 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:09.079583    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/943fb073-b5af-467c-a4fe-16d9a0cb07bc-node-exporter-accelerators-collector-config\") pod \"node-exporter-gc7xw\" (UID: \"943fb073-b5af-467c-a4fe-16d9a0cb07bc\") " pod="openshift-monitoring/node-exporter-gc7xw"
Jul 12 17:04:09.079727 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:09.079601    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/943fb073-b5af-467c-a4fe-16d9a0cb07bc-node-exporter-textfile\") pod \"node-exporter-gc7xw\" (UID: \"943fb073-b5af-467c-a4fe-16d9a0cb07bc\") " pod="openshift-monitoring/node-exporter-gc7xw"
Jul 12 17:04:09.079727 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:09.079618    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/943fb073-b5af-467c-a4fe-16d9a0cb07bc-node-exporter-wtmp\") pod \"node-exporter-gc7xw\" (UID: \"943fb073-b5af-467c-a4fe-16d9a0cb07bc\") " pod="openshift-monitoring/node-exporter-gc7xw"
Jul 12 17:04:09.079727 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:09.079643    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/156026b1-c28b-43ca-9249-6c85d5de032e-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-x46rv\" (UID: \"156026b1-c28b-43ca-9249-6c85d5de032e\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-x46rv"
Jul 12 17:04:09.079727 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:04:09.079655    2569 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 12 17:04:09.079727 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:04:09.079730    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/943fb073-b5af-467c-a4fe-16d9a0cb07bc-node-exporter-tls podName:943fb073-b5af-467c-a4fe-16d9a0cb07bc nodeName:}" failed. No retries permitted until 2026-07-12 17:04:09.579707841 +0000 UTC m=+57.700024641 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/943fb073-b5af-467c-a4fe-16d9a0cb07bc-node-exporter-tls") pod "node-exporter-gc7xw" (UID: "943fb073-b5af-467c-a4fe-16d9a0cb07bc") : secret "node-exporter-tls" not found
Jul 12 17:04:09.080250 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:09.079779    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/943fb073-b5af-467c-a4fe-16d9a0cb07bc-node-exporter-wtmp\") pod \"node-exporter-gc7xw\" (UID: \"943fb073-b5af-467c-a4fe-16d9a0cb07bc\") " pod="openshift-monitoring/node-exporter-gc7xw"
Jul 12 17:04:09.080250 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:04:09.079657    2569 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Jul 12 17:04:09.080250 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:09.079661    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjj8k\" (UniqueName: \"kubernetes.io/projected/156026b1-c28b-43ca-9249-6c85d5de032e-kube-api-access-gjj8k\") pod \"openshift-state-metrics-84bd68b896-x46rv\" (UID: \"156026b1-c28b-43ca-9249-6c85d5de032e\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-x46rv"
Jul 12 17:04:09.080250 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:04:09.079861    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/156026b1-c28b-43ca-9249-6c85d5de032e-openshift-state-metrics-tls podName:156026b1-c28b-43ca-9249-6c85d5de032e nodeName:}" failed. No retries permitted until 2026-07-12 17:04:09.579843224 +0000 UTC m=+57.700160013 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/156026b1-c28b-43ca-9249-6c85d5de032e-openshift-state-metrics-tls") pod "openshift-state-metrics-84bd68b896-x46rv" (UID: "156026b1-c28b-43ca-9249-6c85d5de032e") : secret "openshift-state-metrics-tls" not found
Jul 12 17:04:09.080250 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:09.079880    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/943fb073-b5af-467c-a4fe-16d9a0cb07bc-root\") pod \"node-exporter-gc7xw\" (UID: \"943fb073-b5af-467c-a4fe-16d9a0cb07bc\") " pod="openshift-monitoring/node-exporter-gc7xw"
Jul 12 17:04:09.080250 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:09.079919    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/943fb073-b5af-467c-a4fe-16d9a0cb07bc-sys\") pod \"node-exporter-gc7xw\" (UID: \"943fb073-b5af-467c-a4fe-16d9a0cb07bc\") " pod="openshift-monitoring/node-exporter-gc7xw"
Jul 12 17:04:09.080250 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:09.079951    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/943fb073-b5af-467c-a4fe-16d9a0cb07bc-metrics-client-ca\") pod \"node-exporter-gc7xw\" (UID: \"943fb073-b5af-467c-a4fe-16d9a0cb07bc\") " pod="openshift-monitoring/node-exporter-gc7xw"
Jul 12 17:04:09.080250 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:09.079955    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/943fb073-b5af-467c-a4fe-16d9a0cb07bc-root\") pod \"node-exporter-gc7xw\" (UID: \"943fb073-b5af-467c-a4fe-16d9a0cb07bc\") " pod="openshift-monitoring/node-exporter-gc7xw"
Jul 12 17:04:09.080250 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:09.079983    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sltq9\" (UniqueName: \"kubernetes.io/projected/943fb073-b5af-467c-a4fe-16d9a0cb07bc-kube-api-access-sltq9\") pod \"node-exporter-gc7xw\" (UID: \"943fb073-b5af-467c-a4fe-16d9a0cb07bc\") " pod="openshift-monitoring/node-exporter-gc7xw"
Jul 12 17:04:09.080250 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:09.080011    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/943fb073-b5af-467c-a4fe-16d9a0cb07bc-sys\") pod \"node-exporter-gc7xw\" (UID: \"943fb073-b5af-467c-a4fe-16d9a0cb07bc\") " pod="openshift-monitoring/node-exporter-gc7xw"
Jul 12 17:04:09.080250 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:09.080018    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/156026b1-c28b-43ca-9249-6c85d5de032e-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-x46rv\" (UID: \"156026b1-c28b-43ca-9249-6c85d5de032e\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-x46rv"
Jul 12 17:04:09.080250 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:09.080216    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/943fb073-b5af-467c-a4fe-16d9a0cb07bc-node-exporter-textfile\") pod \"node-exporter-gc7xw\" (UID: \"943fb073-b5af-467c-a4fe-16d9a0cb07bc\") " pod="openshift-monitoring/node-exporter-gc7xw"
Jul 12 17:04:09.080700 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:09.080290    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/943fb073-b5af-467c-a4fe-16d9a0cb07bc-node-exporter-accelerators-collector-config\") pod \"node-exporter-gc7xw\" (UID: \"943fb073-b5af-467c-a4fe-16d9a0cb07bc\") " pod="openshift-monitoring/node-exporter-gc7xw"
Jul 12 17:04:09.080700 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:09.080330    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/156026b1-c28b-43ca-9249-6c85d5de032e-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-x46rv\" (UID: \"156026b1-c28b-43ca-9249-6c85d5de032e\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-x46rv"
Jul 12 17:04:09.080700 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:09.080658    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/943fb073-b5af-467c-a4fe-16d9a0cb07bc-metrics-client-ca\") pod \"node-exporter-gc7xw\" (UID: \"943fb073-b5af-467c-a4fe-16d9a0cb07bc\") " pod="openshift-monitoring/node-exporter-gc7xw"
Jul 12 17:04:09.083482 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:09.083456    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/943fb073-b5af-467c-a4fe-16d9a0cb07bc-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gc7xw\" (UID: \"943fb073-b5af-467c-a4fe-16d9a0cb07bc\") " pod="openshift-monitoring/node-exporter-gc7xw"
Jul 12 17:04:09.083596 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:09.083500    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/156026b1-c28b-43ca-9249-6c85d5de032e-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-x46rv\" (UID: \"156026b1-c28b-43ca-9249-6c85d5de032e\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-x46rv"
Jul 12 17:04:09.089009 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:09.088954    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sltq9\" (UniqueName: \"kubernetes.io/projected/943fb073-b5af-467c-a4fe-16d9a0cb07bc-kube-api-access-sltq9\") pod \"node-exporter-gc7xw\" (UID: \"943fb073-b5af-467c-a4fe-16d9a0cb07bc\") " pod="openshift-monitoring/node-exporter-gc7xw"
Jul 12 17:04:09.090166 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:09.090140    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjj8k\" (UniqueName: \"kubernetes.io/projected/156026b1-c28b-43ca-9249-6c85d5de032e-kube-api-access-gjj8k\") pod \"openshift-state-metrics-84bd68b896-x46rv\" (UID: \"156026b1-c28b-43ca-9249-6c85d5de032e\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-x46rv"
Jul 12 17:04:09.414367 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:09.414270    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-55b6b74c7-5jbs4"
Jul 12 17:04:09.414367 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:09.414356    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-55b6b74c7-5jbs4"
Jul 12 17:04:09.419056 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:09.419032    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-55b6b74c7-5jbs4"
Jul 12 17:04:09.584792 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:09.584756    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/156026b1-c28b-43ca-9249-6c85d5de032e-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-x46rv\" (UID: \"156026b1-c28b-43ca-9249-6c85d5de032e\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-x46rv"
Jul 12 17:04:09.584792 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:09.584801    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/943fb073-b5af-467c-a4fe-16d9a0cb07bc-node-exporter-tls\") pod \"node-exporter-gc7xw\" (UID: \"943fb073-b5af-467c-a4fe-16d9a0cb07bc\") " pod="openshift-monitoring/node-exporter-gc7xw"
Jul 12 17:04:09.587107 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:09.587084    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/943fb073-b5af-467c-a4fe-16d9a0cb07bc-node-exporter-tls\") pod \"node-exporter-gc7xw\" (UID: \"943fb073-b5af-467c-a4fe-16d9a0cb07bc\") " pod="openshift-monitoring/node-exporter-gc7xw"
Jul 12 17:04:09.587218 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:09.587201    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/156026b1-c28b-43ca-9249-6c85d5de032e-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-x46rv\" (UID: \"156026b1-c28b-43ca-9249-6c85d5de032e\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-x46rv"
Jul 12 17:04:09.716273 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:09.716243    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xztsd" event={"ID":"5c7f573f-607f-40cd-b450-1976b2898b83","Type":"ContainerStarted","Data":"01916149e4b07a7d758579d1dc75e6bd47580dfc8629eda14be92c5287afa2de"}
Jul 12 17:04:09.720139 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:09.720118    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-55b6b74c7-5jbs4"
Jul 12 17:04:09.737747 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:09.737701    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-xztsd" podStartSLOduration=1.611663755 podStartE2EDuration="12.737686361s" totalImagesPullingTime="11.126022606s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-12 17:03:57 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 17:04:09.737642192 +0000 UTC m=+57.857958990" watchObservedRunningTime="2026-07-12 17:04:09.737686361 +0000 UTC m=+57.858003157"
Jul 12 17:04:09.813712 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:09.813681    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-x46rv"
Jul 12 17:04:09.827078 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:09.827058    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-gc7xw"
Jul 12 17:04:09.837195 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:04:09.837171    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod943fb073_b5af_467c_a4fe_16d9a0cb07bc.slice/crio-f313c64e067d1f199086aa2ff82a28e613eb1b59574f65a86d602c7e28d13927 WatchSource:0}: Error finding container f313c64e067d1f199086aa2ff82a28e613eb1b59574f65a86d602c7e28d13927: Status 404 returned error can't find the container with id f313c64e067d1f199086aa2ff82a28e613eb1b59574f65a86d602c7e28d13927
Jul 12 17:04:09.954565 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:09.954534    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-x46rv"]
Jul 12 17:04:09.957244 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:04:09.957214    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod156026b1_c28b_43ca_9249_6c85d5de032e.slice/crio-6c3b1641d7899154b1a6b1dcc5f01c488e41e36b977454cc12479b81e3d0779e WatchSource:0}: Error finding container 6c3b1641d7899154b1a6b1dcc5f01c488e41e36b977454cc12479b81e3d0779e: Status 404 returned error can't find the container with id 6c3b1641d7899154b1a6b1dcc5f01c488e41e36b977454cc12479b81e3d0779e
Jul 12 17:04:10.060751 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:10.060722    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 12 17:04:10.063824 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:10.063762    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 12 17:04:10.067074 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:10.067054    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 12 17:04:10.067169 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:10.067054    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 12 17:04:10.067814 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:10.067775    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 12 17:04:10.067814 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:10.067788    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 12 17:04:10.068022 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:10.067839    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-sbl8b\""
Jul 12 17:04:10.068022 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:10.067870    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 12 17:04:10.068122 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:10.068087    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 12 17:04:10.068172 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:10.068140    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 12 17:04:10.068172 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:10.068157    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 12 17:04:10.073041 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:10.073022    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 12 17:04:10.089521 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:10.089487    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3cec2e0b-d740-4361-b911-222469a7e0a8-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"3cec2e0b-d740-4361-b911-222469a7e0a8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 12 17:04:10.089623 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:10.089552    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3cec2e0b-d740-4361-b911-222469a7e0a8-config-out\") pod \"alertmanager-main-0\" (UID: \"3cec2e0b-d740-4361-b911-222469a7e0a8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 12 17:04:10.089623 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:10.089580    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sw92s\" (UniqueName: \"kubernetes.io/projected/3cec2e0b-d740-4361-b911-222469a7e0a8-kube-api-access-sw92s\") pod \"alertmanager-main-0\" (UID: \"3cec2e0b-d740-4361-b911-222469a7e0a8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 12 17:04:10.089623 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:10.089612    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3cec2e0b-d740-4361-b911-222469a7e0a8-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"3cec2e0b-d740-4361-b911-222469a7e0a8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 12 17:04:10.089750 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:10.089649    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/3cec2e0b-d740-4361-b911-222469a7e0a8-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"3cec2e0b-d740-4361-b911-222469a7e0a8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 12 17:04:10.089750 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:10.089702    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3cec2e0b-d740-4361-b911-222469a7e0a8-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"3cec2e0b-d740-4361-b911-222469a7e0a8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 12 17:04:10.089750 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:10.089744    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3cec2e0b-d740-4361-b911-222469a7e0a8-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"3cec2e0b-d740-4361-b911-222469a7e0a8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 12 17:04:10.089891 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:10.089791    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3cec2e0b-d740-4361-b911-222469a7e0a8-tls-assets\") pod \"alertmanager-main-0\" (UID: \"3cec2e0b-d740-4361-b911-222469a7e0a8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 12 17:04:10.089891 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:10.089860    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3cec2e0b-d740-4361-b911-222469a7e0a8-config-volume\") pod \"alertmanager-main-0\" (UID: \"3cec2e0b-d740-4361-b911-222469a7e0a8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 12 17:04:10.089987 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:10.089898    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/3cec2e0b-d740-4361-b911-222469a7e0a8-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"3cec2e0b-d740-4361-b911-222469a7e0a8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 12 17:04:10.089987 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:10.089935    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3cec2e0b-d740-4361-b911-222469a7e0a8-web-config\") pod \"alertmanager-main-0\" (UID: \"3cec2e0b-d740-4361-b911-222469a7e0a8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 12 17:04:10.089987 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:10.089963    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3cec2e0b-d740-4361-b911-222469a7e0a8-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"3cec2e0b-d740-4361-b911-222469a7e0a8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 12 17:04:10.090120 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:10.090010    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3cec2e0b-d740-4361-b911-222469a7e0a8-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"3cec2e0b-d740-4361-b911-222469a7e0a8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 12 17:04:10.096045 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:10.096027    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 12 17:04:10.190736 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:10.190702    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3cec2e0b-d740-4361-b911-222469a7e0a8-config-volume\") pod \"alertmanager-main-0\" (UID: \"3cec2e0b-d740-4361-b911-222469a7e0a8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 12 17:04:10.190891 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:10.190761    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/3cec2e0b-d740-4361-b911-222469a7e0a8-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"3cec2e0b-d740-4361-b911-222469a7e0a8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 12 17:04:10.190891 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:10.190797    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3cec2e0b-d740-4361-b911-222469a7e0a8-web-config\") pod \"alertmanager-main-0\" (UID: \"3cec2e0b-d740-4361-b911-222469a7e0a8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 12 17:04:10.190891 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:10.190821    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3cec2e0b-d740-4361-b911-222469a7e0a8-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"3cec2e0b-d740-4361-b911-222469a7e0a8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 12 17:04:10.190891 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:10.190850    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3cec2e0b-d740-4361-b911-222469a7e0a8-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"3cec2e0b-d740-4361-b911-222469a7e0a8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 12 17:04:10.191102 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:10.190916    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3cec2e0b-d740-4361-b911-222469a7e0a8-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"3cec2e0b-d740-4361-b911-222469a7e0a8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 12 17:04:10.191102 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:10.190952    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3cec2e0b-d740-4361-b911-222469a7e0a8-config-out\") pod \"alertmanager-main-0\" (UID: \"3cec2e0b-d740-4361-b911-222469a7e0a8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 12 17:04:10.191102 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:10.190975    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sw92s\" (UniqueName: \"kubernetes.io/projected/3cec2e0b-d740-4361-b911-222469a7e0a8-kube-api-access-sw92s\") pod \"alertmanager-main-0\" (UID: \"3cec2e0b-d740-4361-b911-222469a7e0a8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 12 17:04:10.191102 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:10.191011    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3cec2e0b-d740-4361-b911-222469a7e0a8-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"3cec2e0b-d740-4361-b911-222469a7e0a8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 12 17:04:10.191102 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:10.191037    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/3cec2e0b-d740-4361-b911-222469a7e0a8-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"3cec2e0b-d740-4361-b911-222469a7e0a8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 12 17:04:10.191102 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:10.191060    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3cec2e0b-d740-4361-b911-222469a7e0a8-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"3cec2e0b-d740-4361-b911-222469a7e0a8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 12 17:04:10.191102 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:10.191088    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3cec2e0b-d740-4361-b911-222469a7e0a8-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"3cec2e0b-d740-4361-b911-222469a7e0a8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 12 17:04:10.191102 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:04:10.191090    2569 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found
Jul 12 17:04:10.191582 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:10.191123    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3cec2e0b-d740-4361-b911-222469a7e0a8-tls-assets\") pod \"alertmanager-main-0\" (UID: \"3cec2e0b-d740-4361-b911-222469a7e0a8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 12 17:04:10.191582 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:04:10.191168    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3cec2e0b-d740-4361-b911-222469a7e0a8-secret-alertmanager-main-tls podName:3cec2e0b-d740-4361-b911-222469a7e0a8 nodeName:}" failed. No retries permitted until 2026-07-12 17:04:10.691140868 +0000 UTC m=+58.811457649 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/3cec2e0b-d740-4361-b911-222469a7e0a8-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "3cec2e0b-d740-4361-b911-222469a7e0a8") : secret "alertmanager-main-tls" not found
Jul 12 17:04:10.191817 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:10.191769    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3cec2e0b-d740-4361-b911-222469a7e0a8-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"3cec2e0b-d740-4361-b911-222469a7e0a8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 12 17:04:10.192012 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:04:10.191993    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3cec2e0b-d740-4361-b911-222469a7e0a8-alertmanager-trusted-ca-bundle podName:3cec2e0b-d740-4361-b911-222469a7e0a8 nodeName:}" failed. No retries permitted until 2026-07-12 17:04:10.69197623 +0000 UTC m=+58.812293008 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/3cec2e0b-d740-4361-b911-222469a7e0a8-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "3cec2e0b-d740-4361-b911-222469a7e0a8") : configmap references non-existent config key: ca-bundle.crt
Jul 12 17:04:10.193776 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:10.193755    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/3cec2e0b-d740-4361-b911-222469a7e0a8-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"3cec2e0b-d740-4361-b911-222469a7e0a8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 12 17:04:10.194032 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:10.193821    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3cec2e0b-d740-4361-b911-222469a7e0a8-tls-assets\") pod \"alertmanager-main-0\" (UID: \"3cec2e0b-d740-4361-b911-222469a7e0a8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 12 17:04:10.195805 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:10.195785    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3cec2e0b-d740-4361-b911-222469a7e0a8-config-volume\") pod \"alertmanager-main-0\" (UID: \"3cec2e0b-d740-4361-b911-222469a7e0a8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 12 17:04:10.196128 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:10.196046    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3cec2e0b-d740-4361-b911-222469a7e0a8-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"3cec2e0b-d740-4361-b911-222469a7e0a8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 12 17:04:10.196128 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:10.196050    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3cec2e0b-d740-4361-b911-222469a7e0a8-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"3cec2e0b-d740-4361-b911-222469a7e0a8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 12 17:04:10.196781 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:10.196757    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3cec2e0b-d740-4361-b911-222469a7e0a8-web-config\") pod \"alertmanager-main-0\" (UID: \"3cec2e0b-d740-4361-b911-222469a7e0a8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 12 17:04:10.197776 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:10.197756    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3cec2e0b-d740-4361-b911-222469a7e0a8-config-out\") pod \"alertmanager-main-0\" (UID: \"3cec2e0b-d740-4361-b911-222469a7e0a8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 12 17:04:10.198385 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:10.198360    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/3cec2e0b-d740-4361-b911-222469a7e0a8-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"3cec2e0b-d740-4361-b911-222469a7e0a8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 12 17:04:10.198599 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:10.198561    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3cec2e0b-d740-4361-b911-222469a7e0a8-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"3cec2e0b-d740-4361-b911-222469a7e0a8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 12 17:04:10.205904 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:10.205865    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sw92s\" (UniqueName: \"kubernetes.io/projected/3cec2e0b-d740-4361-b911-222469a7e0a8-kube-api-access-sw92s\") pod \"alertmanager-main-0\" (UID: \"3cec2e0b-d740-4361-b911-222469a7e0a8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 12 17:04:10.696760 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:10.696720    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3cec2e0b-d740-4361-b911-222469a7e0a8-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"3cec2e0b-d740-4361-b911-222469a7e0a8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 12 17:04:10.696929 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:10.696794    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3cec2e0b-d740-4361-b911-222469a7e0a8-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"3cec2e0b-d740-4361-b911-222469a7e0a8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 12 17:04:10.696929 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:04:10.696853    2569 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found
Jul 12 17:04:10.697016 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:04:10.696935    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3cec2e0b-d740-4361-b911-222469a7e0a8-secret-alertmanager-main-tls podName:3cec2e0b-d740-4361-b911-222469a7e0a8 nodeName:}" failed. No retries permitted until 2026-07-12 17:04:11.696910017 +0000 UTC m=+59.817226796 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/3cec2e0b-d740-4361-b911-222469a7e0a8-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "3cec2e0b-d740-4361-b911-222469a7e0a8") : secret "alertmanager-main-tls" not found
Jul 12 17:04:10.697652 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:10.697629    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3cec2e0b-d740-4361-b911-222469a7e0a8-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"3cec2e0b-d740-4361-b911-222469a7e0a8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 12 17:04:10.722237 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:10.722207    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-x46rv" event={"ID":"156026b1-c28b-43ca-9249-6c85d5de032e","Type":"ContainerStarted","Data":"91b6f55c77c5be968979cf2b282bdaf6604a7100e90ed1d674dd17c54e137be7"}
Jul 12 17:04:10.722352 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:10.722272    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-x46rv" event={"ID":"156026b1-c28b-43ca-9249-6c85d5de032e","Type":"ContainerStarted","Data":"dee85cad3e38c799cd6102f80e488cadea9f174c15229ee72769bc101bcc250e"}
Jul 12 17:04:10.722352 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:10.722306    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-x46rv" event={"ID":"156026b1-c28b-43ca-9249-6c85d5de032e","Type":"ContainerStarted","Data":"6c3b1641d7899154b1a6b1dcc5f01c488e41e36b977454cc12479b81e3d0779e"}
Jul 12 17:04:10.723611 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:10.723564    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gc7xw" event={"ID":"943fb073-b5af-467c-a4fe-16d9a0cb07bc","Type":"ContainerStarted","Data":"f313c64e067d1f199086aa2ff82a28e613eb1b59574f65a86d602c7e28d13927"}
Jul 12 17:04:11.704697 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:11.704664    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3cec2e0b-d740-4361-b911-222469a7e0a8-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"3cec2e0b-d740-4361-b911-222469a7e0a8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 12 17:04:11.706968 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:11.706941    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3cec2e0b-d740-4361-b911-222469a7e0a8-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"3cec2e0b-d740-4361-b911-222469a7e0a8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 12 17:04:11.728548 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:11.728526    2569 generic.go:356] "Generic (PLEG): container finished" podID="943fb073-b5af-467c-a4fe-16d9a0cb07bc" containerID="3544a9097bb2a254f9c426ab68a3d061ca5463386f386d451aed364b0648875f" exitCode=0
Jul 12 17:04:11.728669 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:11.728617    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gc7xw" event={"ID":"943fb073-b5af-467c-a4fe-16d9a0cb07bc","Type":"ContainerDied","Data":"3544a9097bb2a254f9c426ab68a3d061ca5463386f386d451aed364b0648875f"}
Jul 12 17:04:11.730816 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:11.730790    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-x46rv" event={"ID":"156026b1-c28b-43ca-9249-6c85d5de032e","Type":"ContainerStarted","Data":"1f5e47bdc05015fb9afc39e3574c0cb5f0511bc8b4e87583905ecfca7d311683"}
Jul 12 17:04:11.769752 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:11.769713    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-x46rv" podStartSLOduration=2.974874201 podStartE2EDuration="3.7696995s" totalImagesPullingTime="794.825299ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 17:04:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 17:04:11.768540714 +0000 UTC m=+59.888857511" watchObservedRunningTime="2026-07-12 17:04:11.7696995 +0000 UTC m=+59.890016297"
Jul 12 17:04:11.873071 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:11.873036    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 12 17:04:12.018688 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:12.018615    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 12 17:04:12.021236 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:04:12.021206    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cec2e0b_d740_4361_b911_222469a7e0a8.slice/crio-3b92a00d3355f6226378f04146b995c82023e5797f883e6489b9dee5422cb9b1 WatchSource:0}: Error finding container 3b92a00d3355f6226378f04146b995c82023e5797f883e6489b9dee5422cb9b1: Status 404 returned error can't find the container with id 3b92a00d3355f6226378f04146b995c82023e5797f883e6489b9dee5422cb9b1
Jul 12 17:04:12.633132 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:12.633102    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9xbrj"
Jul 12 17:04:12.737369 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:12.737331    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gc7xw" event={"ID":"943fb073-b5af-467c-a4fe-16d9a0cb07bc","Type":"ContainerStarted","Data":"5fa5e9602cf205d59c682f0f24935863e76b1c110122b6e6ab9aee1c79283d43"}
Jul 12 17:04:12.737830 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:12.737403    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gc7xw" event={"ID":"943fb073-b5af-467c-a4fe-16d9a0cb07bc","Type":"ContainerStarted","Data":"ab9a7c177be98cb1db00212fd83e7abba7776c9ce223ba7e6dd7ef055cedb569"}
Jul 12 17:04:12.738563 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:12.738540    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3cec2e0b-d740-4361-b911-222469a7e0a8","Type":"ContainerStarted","Data":"3b92a00d3355f6226378f04146b995c82023e5797f883e6489b9dee5422cb9b1"}
Jul 12 17:04:12.758858 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:12.758806    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-gc7xw" podStartSLOduration=3.680595367 podStartE2EDuration="4.758786135s" totalImagesPullingTime="1.012406888s" totalInitContainerRuntime="65.78388ms" isStatefulPod=true podCreationTimestamp="2026-07-12 17:04:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 17:04:12.757944904 +0000 UTC m=+60.878261701" watchObservedRunningTime="2026-07-12 17:04:12.758786135 +0000 UTC m=+60.879102934"
Jul 12 17:04:13.199609 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:13.199569    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-5866f48f55-vmnx5"]
Jul 12 17:04:13.209956 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:13.209929    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5866f48f55-vmnx5"
Jul 12 17:04:13.212473 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:13.212416    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5866f48f55-vmnx5"]
Jul 12 17:04:13.212833 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:13.212810    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Jul 12 17:04:13.212953 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:13.212858    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-jnsrk\""
Jul 12 17:04:13.212953 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:13.212810    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Jul 12 17:04:13.212953 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:13.212887    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 12 17:04:13.212953 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:13.212902    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Jul 12 17:04:13.213124 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:13.213026    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-2psl27hb1fklb\""
Jul 12 17:04:13.318351 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:13.318324    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/63874cd4-7f4d-4834-9e6b-1e524494498a-metrics-server-audit-profiles\") pod \"metrics-server-5866f48f55-vmnx5\" (UID: \"63874cd4-7f4d-4834-9e6b-1e524494498a\") " pod="openshift-monitoring/metrics-server-5866f48f55-vmnx5"
Jul 12 17:04:13.318475 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:13.318364    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/63874cd4-7f4d-4834-9e6b-1e524494498a-client-ca-bundle\") pod \"metrics-server-5866f48f55-vmnx5\" (UID: \"63874cd4-7f4d-4834-9e6b-1e524494498a\") " pod="openshift-monitoring/metrics-server-5866f48f55-vmnx5"
Jul 12 17:04:13.318475 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:13.318390    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/63874cd4-7f4d-4834-9e6b-1e524494498a-secret-metrics-server-tls\") pod \"metrics-server-5866f48f55-vmnx5\" (UID: \"63874cd4-7f4d-4834-9e6b-1e524494498a\") " pod="openshift-monitoring/metrics-server-5866f48f55-vmnx5"
Jul 12 17:04:13.318475 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:13.318408    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/63874cd4-7f4d-4834-9e6b-1e524494498a-secret-metrics-server-client-certs\") pod \"metrics-server-5866f48f55-vmnx5\" (UID: \"63874cd4-7f4d-4834-9e6b-1e524494498a\") " pod="openshift-monitoring/metrics-server-5866f48f55-vmnx5"
Jul 12 17:04:13.318475 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:13.318426    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/63874cd4-7f4d-4834-9e6b-1e524494498a-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5866f48f55-vmnx5\" (UID: \"63874cd4-7f4d-4834-9e6b-1e524494498a\") " pod="openshift-monitoring/metrics-server-5866f48f55-vmnx5"
Jul 12 17:04:13.318700 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:13.318492    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/63874cd4-7f4d-4834-9e6b-1e524494498a-audit-log\") pod \"metrics-server-5866f48f55-vmnx5\" (UID: \"63874cd4-7f4d-4834-9e6b-1e524494498a\") " pod="openshift-monitoring/metrics-server-5866f48f55-vmnx5"
Jul 12 17:04:13.318700 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:13.318605    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9hjf\" (UniqueName: \"kubernetes.io/projected/63874cd4-7f4d-4834-9e6b-1e524494498a-kube-api-access-l9hjf\") pod \"metrics-server-5866f48f55-vmnx5\" (UID: \"63874cd4-7f4d-4834-9e6b-1e524494498a\") " pod="openshift-monitoring/metrics-server-5866f48f55-vmnx5"
Jul 12 17:04:13.419425 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:13.419398    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/63874cd4-7f4d-4834-9e6b-1e524494498a-metrics-server-audit-profiles\") pod \"metrics-server-5866f48f55-vmnx5\" (UID: \"63874cd4-7f4d-4834-9e6b-1e524494498a\") " pod="openshift-monitoring/metrics-server-5866f48f55-vmnx5"
Jul 12 17:04:13.419540 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:13.419434    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/63874cd4-7f4d-4834-9e6b-1e524494498a-client-ca-bundle\") pod \"metrics-server-5866f48f55-vmnx5\" (UID: \"63874cd4-7f4d-4834-9e6b-1e524494498a\") " pod="openshift-monitoring/metrics-server-5866f48f55-vmnx5"
Jul 12 17:04:13.419540 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:13.419460    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/63874cd4-7f4d-4834-9e6b-1e524494498a-secret-metrics-server-tls\") pod \"metrics-server-5866f48f55-vmnx5\" (UID: \"63874cd4-7f4d-4834-9e6b-1e524494498a\") " pod="openshift-monitoring/metrics-server-5866f48f55-vmnx5"
Jul 12 17:04:13.419540 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:13.419487    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/63874cd4-7f4d-4834-9e6b-1e524494498a-secret-metrics-server-client-certs\") pod \"metrics-server-5866f48f55-vmnx5\" (UID: \"63874cd4-7f4d-4834-9e6b-1e524494498a\") " pod="openshift-monitoring/metrics-server-5866f48f55-vmnx5"
Jul 12 17:04:13.419540 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:13.419523    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/63874cd4-7f4d-4834-9e6b-1e524494498a-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5866f48f55-vmnx5\" (UID: \"63874cd4-7f4d-4834-9e6b-1e524494498a\") " pod="openshift-monitoring/metrics-server-5866f48f55-vmnx5"
Jul 12 17:04:13.419741 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:13.419544    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/63874cd4-7f4d-4834-9e6b-1e524494498a-audit-log\") pod \"metrics-server-5866f48f55-vmnx5\" (UID: \"63874cd4-7f4d-4834-9e6b-1e524494498a\") " pod="openshift-monitoring/metrics-server-5866f48f55-vmnx5"
Jul 12 17:04:13.419741 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:13.419569    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9hjf\" (UniqueName: \"kubernetes.io/projected/63874cd4-7f4d-4834-9e6b-1e524494498a-kube-api-access-l9hjf\") pod \"metrics-server-5866f48f55-vmnx5\" (UID: \"63874cd4-7f4d-4834-9e6b-1e524494498a\") " pod="openshift-monitoring/metrics-server-5866f48f55-vmnx5"
Jul 12 17:04:13.419987 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:13.419960    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/63874cd4-7f4d-4834-9e6b-1e524494498a-audit-log\") pod \"metrics-server-5866f48f55-vmnx5\" (UID: \"63874cd4-7f4d-4834-9e6b-1e524494498a\") " pod="openshift-monitoring/metrics-server-5866f48f55-vmnx5"
Jul 12 17:04:13.420276 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:13.420253    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/63874cd4-7f4d-4834-9e6b-1e524494498a-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5866f48f55-vmnx5\" (UID: \"63874cd4-7f4d-4834-9e6b-1e524494498a\") " pod="openshift-monitoring/metrics-server-5866f48f55-vmnx5"
Jul 12 17:04:13.420491 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:13.420474    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/63874cd4-7f4d-4834-9e6b-1e524494498a-metrics-server-audit-profiles\") pod \"metrics-server-5866f48f55-vmnx5\" (UID: \"63874cd4-7f4d-4834-9e6b-1e524494498a\") " pod="openshift-monitoring/metrics-server-5866f48f55-vmnx5"
Jul 12 17:04:13.421762 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:13.421745    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/63874cd4-7f4d-4834-9e6b-1e524494498a-secret-metrics-server-client-certs\") pod \"metrics-server-5866f48f55-vmnx5\" (UID: \"63874cd4-7f4d-4834-9e6b-1e524494498a\") " pod="openshift-monitoring/metrics-server-5866f48f55-vmnx5"
Jul 12 17:04:13.422155 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:13.422136    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/63874cd4-7f4d-4834-9e6b-1e524494498a-secret-metrics-server-tls\") pod \"metrics-server-5866f48f55-vmnx5\" (UID: \"63874cd4-7f4d-4834-9e6b-1e524494498a\") " pod="openshift-monitoring/metrics-server-5866f48f55-vmnx5"
Jul 12 17:04:13.422306 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:13.422286    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/63874cd4-7f4d-4834-9e6b-1e524494498a-client-ca-bundle\") pod \"metrics-server-5866f48f55-vmnx5\" (UID: \"63874cd4-7f4d-4834-9e6b-1e524494498a\") " pod="openshift-monitoring/metrics-server-5866f48f55-vmnx5"
Jul 12 17:04:13.428647 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:13.428623    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9hjf\" (UniqueName: \"kubernetes.io/projected/63874cd4-7f4d-4834-9e6b-1e524494498a-kube-api-access-l9hjf\") pod \"metrics-server-5866f48f55-vmnx5\" (UID: \"63874cd4-7f4d-4834-9e6b-1e524494498a\") " pod="openshift-monitoring/metrics-server-5866f48f55-vmnx5"
Jul 12 17:04:13.520667 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:13.520616    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5866f48f55-vmnx5"
Jul 12 17:04:13.704827 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:13.704792    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-862s7"]
Jul 12 17:04:13.707616 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:13.707595    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-862s7"
Jul 12 17:04:13.718976 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:13.718961    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-kn2kq\""
Jul 12 17:04:13.719114 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:13.719098    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Jul 12 17:04:13.721452 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:13.721430    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5866f48f55-vmnx5"]
Jul 12 17:04:13.724437 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:04:13.724414    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63874cd4_7f4d_4834_9e6b_1e524494498a.slice/crio-24df4b7cd4f4bd03b859607752f3398df3e17966347a9d3ea98467c9c4583111 WatchSource:0}: Error finding container 24df4b7cd4f4bd03b859607752f3398df3e17966347a9d3ea98467c9c4583111: Status 404 returned error can't find the container with id 24df4b7cd4f4bd03b859607752f3398df3e17966347a9d3ea98467c9c4583111
Jul 12 17:04:13.742640 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:13.742603    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5866f48f55-vmnx5" event={"ID":"63874cd4-7f4d-4834-9e6b-1e524494498a","Type":"ContainerStarted","Data":"24df4b7cd4f4bd03b859607752f3398df3e17966347a9d3ea98467c9c4583111"}
Jul 12 17:04:13.744382 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:13.744361    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-862s7"]
Jul 12 17:04:13.744464 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:13.744408    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3cec2e0b-d740-4361-b911-222469a7e0a8","Type":"ContainerDied","Data":"718f703a53ceff3baa84fe3823cdd2cfb380983827b9dd70bdae0cb7c205c364"}
Jul 12 17:04:13.744464 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:13.744360    2569 generic.go:356] "Generic (PLEG): container finished" podID="3cec2e0b-d740-4361-b911-222469a7e0a8" containerID="718f703a53ceff3baa84fe3823cdd2cfb380983827b9dd70bdae0cb7c205c364" exitCode=0
Jul 12 17:04:13.785265 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:13.785208    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5ff95b89fb-ccqfl"]
Jul 12 17:04:13.787409 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:13.787391    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5ff95b89fb-ccqfl"
Jul 12 17:04:13.796524 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:13.796490    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 12 17:04:13.803010 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:13.802989    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5ff95b89fb-ccqfl"]
Jul 12 17:04:13.823203 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:13.823179    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqkvh\" (UniqueName: \"kubernetes.io/projected/b4d58845-a8df-44ff-ada6-9e794ad37f51-kube-api-access-fqkvh\") pod \"console-5ff95b89fb-ccqfl\" (UID: \"b4d58845-a8df-44ff-ada6-9e794ad37f51\") " pod="openshift-console/console-5ff95b89fb-ccqfl"
Jul 12 17:04:13.823292 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:13.823218    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b4d58845-a8df-44ff-ada6-9e794ad37f51-oauth-serving-cert\") pod \"console-5ff95b89fb-ccqfl\" (UID: \"b4d58845-a8df-44ff-ada6-9e794ad37f51\") " pod="openshift-console/console-5ff95b89fb-ccqfl"
Jul 12 17:04:13.823338 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:13.823296    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tf6w5\" (UniqueName: \"kubernetes.io/projected/dcf4607e-6cb3-4f9c-b3b3-9e2245bc8867-kube-api-access-tf6w5\") pod \"monitoring-plugin-5f68df7f58-862s7\" (UID: \"dcf4607e-6cb3-4f9c-b3b3-9e2245bc8867\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-862s7"
Jul 12 17:04:13.823378 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:13.823335    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/dcf4607e-6cb3-4f9c-b3b3-9e2245bc8867-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-862s7\" (UID: \"dcf4607e-6cb3-4f9c-b3b3-9e2245bc8867\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-862s7"
Jul 12 17:04:13.823378 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:13.823358    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4d58845-a8df-44ff-ada6-9e794ad37f51-trusted-ca-bundle\") pod \"console-5ff95b89fb-ccqfl\" (UID: \"b4d58845-a8df-44ff-ada6-9e794ad37f51\") " pod="openshift-console/console-5ff95b89fb-ccqfl"
Jul 12 17:04:13.823726 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:13.823703    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b4d58845-a8df-44ff-ada6-9e794ad37f51-service-ca\") pod \"console-5ff95b89fb-ccqfl\" (UID: \"b4d58845-a8df-44ff-ada6-9e794ad37f51\") " pod="openshift-console/console-5ff95b89fb-ccqfl"
Jul 12 17:04:13.823795 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:13.823777    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b4d58845-a8df-44ff-ada6-9e794ad37f51-console-serving-cert\") pod \"console-5ff95b89fb-ccqfl\" (UID: \"b4d58845-a8df-44ff-ada6-9e794ad37f51\") " pod="openshift-console/console-5ff95b89fb-ccqfl"
Jul 12 17:04:13.823851 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:13.823809    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b4d58845-a8df-44ff-ada6-9e794ad37f51-console-config\") pod \"console-5ff95b89fb-ccqfl\" (UID: \"b4d58845-a8df-44ff-ada6-9e794ad37f51\") " pod="openshift-console/console-5ff95b89fb-ccqfl"
Jul 12 17:04:13.823939 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:13.823902    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b4d58845-a8df-44ff-ada6-9e794ad37f51-console-oauth-config\") pod \"console-5ff95b89fb-ccqfl\" (UID: \"b4d58845-a8df-44ff-ada6-9e794ad37f51\") " pod="openshift-console/console-5ff95b89fb-ccqfl"
Jul 12 17:04:13.924478 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:13.924448    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqkvh\" (UniqueName: \"kubernetes.io/projected/b4d58845-a8df-44ff-ada6-9e794ad37f51-kube-api-access-fqkvh\") pod \"console-5ff95b89fb-ccqfl\" (UID: \"b4d58845-a8df-44ff-ada6-9e794ad37f51\") " pod="openshift-console/console-5ff95b89fb-ccqfl"
Jul 12 17:04:13.924622 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:13.924484    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b4d58845-a8df-44ff-ada6-9e794ad37f51-oauth-serving-cert\") pod \"console-5ff95b89fb-ccqfl\" (UID: \"b4d58845-a8df-44ff-ada6-9e794ad37f51\") " pod="openshift-console/console-5ff95b89fb-ccqfl"
Jul 12 17:04:13.924622 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:13.924521    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tf6w5\" (UniqueName: \"kubernetes.io/projected/dcf4607e-6cb3-4f9c-b3b3-9e2245bc8867-kube-api-access-tf6w5\") pod \"monitoring-plugin-5f68df7f58-862s7\" (UID: \"dcf4607e-6cb3-4f9c-b3b3-9e2245bc8867\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-862s7"
Jul 12 17:04:13.924753 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:13.924663    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/dcf4607e-6cb3-4f9c-b3b3-9e2245bc8867-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-862s7\" (UID: \"dcf4607e-6cb3-4f9c-b3b3-9e2245bc8867\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-862s7"
Jul 12 17:04:13.924753 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:13.924698    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4d58845-a8df-44ff-ada6-9e794ad37f51-trusted-ca-bundle\") pod \"console-5ff95b89fb-ccqfl\" (UID: \"b4d58845-a8df-44ff-ada6-9e794ad37f51\") " pod="openshift-console/console-5ff95b89fb-ccqfl"
Jul 12 17:04:13.924848 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:04:13.924779    2569 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Jul 12 17:04:13.924848 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:13.924779    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b4d58845-a8df-44ff-ada6-9e794ad37f51-service-ca\") pod \"console-5ff95b89fb-ccqfl\" (UID: \"b4d58845-a8df-44ff-ada6-9e794ad37f51\") " pod="openshift-console/console-5ff95b89fb-ccqfl"
Jul 12 17:04:13.924848 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:04:13.924835    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dcf4607e-6cb3-4f9c-b3b3-9e2245bc8867-monitoring-plugin-cert podName:dcf4607e-6cb3-4f9c-b3b3-9e2245bc8867 nodeName:}" failed. No retries permitted until 2026-07-12 17:04:14.424817073 +0000 UTC m=+62.545133853 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/dcf4607e-6cb3-4f9c-b3b3-9e2245bc8867-monitoring-plugin-cert") pod "monitoring-plugin-5f68df7f58-862s7" (UID: "dcf4607e-6cb3-4f9c-b3b3-9e2245bc8867") : secret "monitoring-plugin-cert" not found
Jul 12 17:04:13.925000 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:13.924871    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b4d58845-a8df-44ff-ada6-9e794ad37f51-console-serving-cert\") pod \"console-5ff95b89fb-ccqfl\" (UID: \"b4d58845-a8df-44ff-ada6-9e794ad37f51\") " pod="openshift-console/console-5ff95b89fb-ccqfl"
Jul 12 17:04:13.925000 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:13.924894    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b4d58845-a8df-44ff-ada6-9e794ad37f51-console-config\") pod \"console-5ff95b89fb-ccqfl\" (UID: \"b4d58845-a8df-44ff-ada6-9e794ad37f51\") " pod="openshift-console/console-5ff95b89fb-ccqfl"
Jul 12 17:04:13.925000 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:13.924928    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b4d58845-a8df-44ff-ada6-9e794ad37f51-console-oauth-config\") pod \"console-5ff95b89fb-ccqfl\" (UID: \"b4d58845-a8df-44ff-ada6-9e794ad37f51\") " pod="openshift-console/console-5ff95b89fb-ccqfl"
Jul 12 17:04:13.925193 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:13.925168    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b4d58845-a8df-44ff-ada6-9e794ad37f51-oauth-serving-cert\") pod \"console-5ff95b89fb-ccqfl\" (UID: \"b4d58845-a8df-44ff-ada6-9e794ad37f51\") " pod="openshift-console/console-5ff95b89fb-ccqfl"
Jul 12 17:04:13.925423 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:13.925399    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b4d58845-a8df-44ff-ada6-9e794ad37f51-service-ca\") pod \"console-5ff95b89fb-ccqfl\" (UID: \"b4d58845-a8df-44ff-ada6-9e794ad37f51\") " pod="openshift-console/console-5ff95b89fb-ccqfl"
Jul 12 17:04:13.925648 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:13.925630    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b4d58845-a8df-44ff-ada6-9e794ad37f51-console-config\") pod \"console-5ff95b89fb-ccqfl\" (UID: \"b4d58845-a8df-44ff-ada6-9e794ad37f51\") " pod="openshift-console/console-5ff95b89fb-ccqfl"
Jul 12 17:04:13.925748 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:13.925726    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4d58845-a8df-44ff-ada6-9e794ad37f51-trusted-ca-bundle\") pod \"console-5ff95b89fb-ccqfl\" (UID: \"b4d58845-a8df-44ff-ada6-9e794ad37f51\") " pod="openshift-console/console-5ff95b89fb-ccqfl"
Jul 12 17:04:13.927242 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:13.927219    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b4d58845-a8df-44ff-ada6-9e794ad37f51-console-oauth-config\") pod \"console-5ff95b89fb-ccqfl\" (UID: \"b4d58845-a8df-44ff-ada6-9e794ad37f51\") " pod="openshift-console/console-5ff95b89fb-ccqfl"
Jul 12 17:04:13.927350 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:13.927299    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b4d58845-a8df-44ff-ada6-9e794ad37f51-console-serving-cert\") pod \"console-5ff95b89fb-ccqfl\" (UID: \"b4d58845-a8df-44ff-ada6-9e794ad37f51\") " pod="openshift-console/console-5ff95b89fb-ccqfl"
Jul 12 17:04:13.932289 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:13.932269    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqkvh\" (UniqueName: \"kubernetes.io/projected/b4d58845-a8df-44ff-ada6-9e794ad37f51-kube-api-access-fqkvh\") pod \"console-5ff95b89fb-ccqfl\" (UID: \"b4d58845-a8df-44ff-ada6-9e794ad37f51\") " pod="openshift-console/console-5ff95b89fb-ccqfl"
Jul 12 17:04:13.932396 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:13.932379    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tf6w5\" (UniqueName: \"kubernetes.io/projected/dcf4607e-6cb3-4f9c-b3b3-9e2245bc8867-kube-api-access-tf6w5\") pod \"monitoring-plugin-5f68df7f58-862s7\" (UID: \"dcf4607e-6cb3-4f9c-b3b3-9e2245bc8867\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-862s7"
Jul 12 17:04:14.096995 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:14.096933    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5ff95b89fb-ccqfl"
Jul 12 17:04:14.244257 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:14.244177    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5ff95b89fb-ccqfl"]
Jul 12 17:04:14.248108 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:04:14.247661    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4d58845_a8df_44ff_ada6_9e794ad37f51.slice/crio-7ae1250091df3b811a4c2f42151c093a145b035dbc750f12231c01e4bd473625 WatchSource:0}: Error finding container 7ae1250091df3b811a4c2f42151c093a145b035dbc750f12231c01e4bd473625: Status 404 returned error can't find the container with id 7ae1250091df3b811a4c2f42151c093a145b035dbc750f12231c01e4bd473625
Jul 12 17:04:14.428120 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:14.428094    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/dcf4607e-6cb3-4f9c-b3b3-9e2245bc8867-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-862s7\" (UID: \"dcf4607e-6cb3-4f9c-b3b3-9e2245bc8867\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-862s7"
Jul 12 17:04:14.430735 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:14.430709    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/dcf4607e-6cb3-4f9c-b3b3-9e2245bc8867-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-862s7\" (UID: \"dcf4607e-6cb3-4f9c-b3b3-9e2245bc8867\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-862s7"
Jul 12 17:04:14.616113 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:14.616079    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-862s7"
Jul 12 17:04:14.756010 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:14.755926    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5ff95b89fb-ccqfl" event={"ID":"b4d58845-a8df-44ff-ada6-9e794ad37f51","Type":"ContainerStarted","Data":"3ccb29edf5f2c04eb590b42afe6949d4a28420b03011a2b20a5a295c5a0e98c0"}
Jul 12 17:04:14.756444 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:14.756008    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5ff95b89fb-ccqfl" event={"ID":"b4d58845-a8df-44ff-ada6-9e794ad37f51","Type":"ContainerStarted","Data":"7ae1250091df3b811a4c2f42151c093a145b035dbc750f12231c01e4bd473625"}
Jul 12 17:04:14.775047 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:14.775005    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5ff95b89fb-ccqfl" podStartSLOduration=1.774990674 podStartE2EDuration="1.774990674s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 17:04:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 17:04:14.774819092 +0000 UTC m=+62.895135893" watchObservedRunningTime="2026-07-12 17:04:14.774990674 +0000 UTC m=+62.895307470"
Jul 12 17:04:15.171536 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:15.171489    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 12 17:04:15.176916 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:15.176886    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 12 17:04:15.179224 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:15.179192    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 12 17:04:15.181910 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:15.179764    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 12 17:04:15.181910 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:15.180167    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 12 17:04:15.181910 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:15.180371    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-62xzw\""
Jul 12 17:04:15.181910 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:15.180703    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 12 17:04:15.181910 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:15.181290    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 12 17:04:15.181910 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:15.181327    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 12 17:04:15.181910 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:15.181597    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-3hs5rhd03u095\""
Jul 12 17:04:15.181910 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:15.181913    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 12 17:04:15.182410 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:15.181992    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 12 17:04:15.182410 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:15.182015    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 12 17:04:15.182410 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:15.182093    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 12 17:04:15.182410 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:15.182391    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 12 17:04:15.182591 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:15.182414    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 12 17:04:15.182591 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:15.182586    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 12 17:04:15.183788 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:15.183763    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 12 17:04:15.198416 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:15.196364    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 12 17:04:15.237266 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:15.237158    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 12 17:04:15.237266 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:15.237205    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea-config-out\") pod \"prometheus-k8s-0\" (UID: \"80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 12 17:04:15.237266 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:15.237235    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 12 17:04:15.237560 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:15.237277    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 12 17:04:15.237560 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:15.237320    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 12 17:04:15.237560 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:15.237350    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 12 17:04:15.237560 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:15.237374    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 12 17:04:15.237560 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:15.237409    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 12 17:04:15.237560 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:15.237431    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 12 17:04:15.237560 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:15.237457    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 12 17:04:15.237560 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:15.237525    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea-web-config\") pod \"prometheus-k8s-0\" (UID: \"80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 12 17:04:15.237560 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:15.237562    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxfwl\" (UniqueName: \"kubernetes.io/projected/80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea-kube-api-access-mxfwl\") pod \"prometheus-k8s-0\" (UID: \"80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 12 17:04:15.237964 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:15.237588    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 12 17:04:15.237964 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:15.237619    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 12 17:04:15.237964 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:15.237643    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 12 17:04:15.237964 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:15.237675    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 12 17:04:15.237964 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:15.237701    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 12 17:04:15.237964 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:15.237729    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea-config\") pod \"prometheus-k8s-0\" (UID: \"80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 12 17:04:15.237964 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:15.237750    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 12 17:04:15.237964 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:15.237770    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 12 17:04:15.339097 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:15.339047    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea-web-config\") pod \"prometheus-k8s-0\" (UID: \"80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 12 17:04:15.339266 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:15.339102    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxfwl\" (UniqueName: \"kubernetes.io/projected/80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea-kube-api-access-mxfwl\") pod \"prometheus-k8s-0\" (UID: \"80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 12 17:04:15.339266 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:15.339131    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 12 17:04:15.339266 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:15.339159    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 12 17:04:15.339266 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:15.339182    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 12 17:04:15.339266 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:15.339219    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 12 17:04:15.339577 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:15.339249    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 12 17:04:15.339577 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:15.339321    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea-config\") pod \"prometheus-k8s-0\" (UID: \"80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 12 17:04:15.339577 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:15.339339    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 12 17:04:15.339577 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:15.339393    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 12 17:04:15.339577 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:15.339441    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 12 17:04:15.339577 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:15.339473    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea-config-out\") pod \"prometheus-k8s-0\" (UID: \"80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 12 17:04:15.339577 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:15.339498    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 12 17:04:15.339577 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:15.339571    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 12 17:04:15.339976 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:15.339620    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 12 17:04:15.339976 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:15.339653    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 12 17:04:15.339976 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:15.339680    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 12 17:04:15.339976 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:15.339719    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 12 17:04:15.339976 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:15.339745    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 12 17:04:15.339976 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:15.339773    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 12 17:04:15.340700 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:15.340633    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 12 17:04:15.341778 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:15.341433    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 12 17:04:15.343213 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:15.343112    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 12 17:04:15.344457 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:15.343491    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 12 17:04:15.344457 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:15.343590    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 12 17:04:15.344457 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:15.343874    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 12 17:04:15.344457 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:15.343938    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 12 17:04:15.344739 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:15.344603    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 12 17:04:15.344849 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:15.344829    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea-config\") pod \"prometheus-k8s-0\" (UID: \"80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 12 17:04:15.344925 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:15.344905    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 12 17:04:15.345142 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:15.345119    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 12 17:04:15.345613 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:15.345593    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 12 17:04:15.345700 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:15.345624    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea-web-config\") pod \"prometheus-k8s-0\" (UID: \"80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 12 17:04:15.346009 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:15.345990    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 12 17:04:15.346754 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:15.346561    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 12 17:04:15.346754 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:15.346714    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 12 17:04:15.346914 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:15.346891    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea-config-out\") pod \"prometheus-k8s-0\" (UID: \"80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 12 17:04:15.346984 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:15.346896    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 12 17:04:15.348642 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:15.348540    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 12 17:04:15.351728 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:15.351708    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxfwl\" (UniqueName: \"kubernetes.io/projected/80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea-kube-api-access-mxfwl\") pod \"prometheus-k8s-0\" (UID: \"80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 12 17:04:15.492892 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:15.492382    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 12 17:04:15.600194 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:15.600166    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-862s7"]
Jul 12 17:04:15.675566 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:15.675544    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 12 17:04:15.761522 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:15.761470    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5866f48f55-vmnx5" event={"ID":"63874cd4-7f4d-4834-9e6b-1e524494498a","Type":"ContainerStarted","Data":"eed9f441a4364f85b3ec64986f250246c5c74110b7f7d2bbf1452aae10cf8972"}
Jul 12 17:04:15.763063 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:15.763037    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea","Type":"ContainerStarted","Data":"c7a9d708b26bace74a6060fc9a225c798971d4747c63cd4c540571c92222036d"}
Jul 12 17:04:15.763153 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:15.763095    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea","Type":"ContainerStarted","Data":"f4fe3dbde4dcc094953caa40ef5b018c1b6d3e945c805b24863b25bc48c62eeb"}
Jul 12 17:04:15.764777 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:15.764744    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-862s7" event={"ID":"dcf4607e-6cb3-4f9c-b3b3-9e2245bc8867","Type":"ContainerStarted","Data":"a8c69d83576e547236610bf3ad895dfbdf84487fa134db1098186fb83606a8a9"}
Jul 12 17:04:15.769064 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:15.769040    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3cec2e0b-d740-4361-b911-222469a7e0a8","Type":"ContainerStarted","Data":"84664ee66a8f4c2fe0824b1c2a99440723a48f9c8d6e0e06ba71c731c973f0e3"}
Jul 12 17:04:15.769260 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:15.769236    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3cec2e0b-d740-4361-b911-222469a7e0a8","Type":"ContainerStarted","Data":"51dd575b946a722aa4bd01f79c91647b4c3bdfaabf6ea3248ac22695b653933f"}
Jul 12 17:04:15.769366 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:15.769285    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3cec2e0b-d740-4361-b911-222469a7e0a8","Type":"ContainerStarted","Data":"f8ea0483a2b54598b42de1bd70aa1a8e0c2f6bc13c7a360099b51306ac3fd594"}
Jul 12 17:04:15.769366 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:15.769327    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3cec2e0b-d740-4361-b911-222469a7e0a8","Type":"ContainerStarted","Data":"ee9b58944010e534f99a91c97f2febe666a9b23d85a88877f476b77c77bbca35"}
Jul 12 17:04:15.783825 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:15.781607    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-5866f48f55-vmnx5" podStartSLOduration=1.0724518 podStartE2EDuration="2.781586025s" totalImagesPullingTime="1.709134225s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 17:04:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 17:04:15.77904962 +0000 UTC m=+63.899366418" watchObservedRunningTime="2026-07-12 17:04:15.781586025 +0000 UTC m=+63.901902823"
Jul 12 17:04:16.711103 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:16.711075    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-2hvcl"
Jul 12 17:04:16.774340 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:16.774312    2569 generic.go:356] "Generic (PLEG): container finished" podID="80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea" containerID="c7a9d708b26bace74a6060fc9a225c798971d4747c63cd4c540571c92222036d" exitCode=0
Jul 12 17:04:16.774724 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:16.774394    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea","Type":"ContainerDied","Data":"c7a9d708b26bace74a6060fc9a225c798971d4747c63cd4c540571c92222036d"}
Jul 12 17:04:16.778068 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:16.778045    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3cec2e0b-d740-4361-b911-222469a7e0a8","Type":"ContainerStarted","Data":"9a127ff898b7d40306157481a7e12303476ac05ef2d4cd1671e126cf4747ed93"}
Jul 12 17:04:17.060673 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:17.060597    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/577e3a93-0560-4099-bddb-dd55563d002d-original-pull-secret\") pod \"global-pull-secret-syncer-mkw2b\" (UID: \"577e3a93-0560-4099-bddb-dd55563d002d\") " pod="kube-system/global-pull-secret-syncer-mkw2b"
Jul 12 17:04:17.060673 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:17.060646    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1d6c71b-d9fb-4f2a-9a81-5e8f0f8a6c13-metrics-certs\") pod \"network-metrics-daemon-h442x\" (UID: \"f1d6c71b-d9fb-4f2a-9a81-5e8f0f8a6c13\") " pod="openshift-multus/network-metrics-daemon-h442x"
Jul 12 17:04:17.063065 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:17.063048    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 12 17:04:17.063112 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:17.063062    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 12 17:04:17.073021 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:17.072991    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1d6c71b-d9fb-4f2a-9a81-5e8f0f8a6c13-metrics-certs\") pod \"network-metrics-daemon-h442x\" (UID: \"f1d6c71b-d9fb-4f2a-9a81-5e8f0f8a6c13\") " pod="openshift-multus/network-metrics-daemon-h442x"
Jul 12 17:04:17.073128 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:17.073035    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/577e3a93-0560-4099-bddb-dd55563d002d-original-pull-secret\") pod \"global-pull-secret-syncer-mkw2b\" (UID: \"577e3a93-0560-4099-bddb-dd55563d002d\") " pod="kube-system/global-pull-secret-syncer-mkw2b"
Jul 12 17:04:17.161684 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:17.161653    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6sf9\" (UniqueName: \"kubernetes.io/projected/259ec8ed-10bd-4fae-8f7f-97409bd4f54b-kube-api-access-n6sf9\") pod \"network-check-target-gxtt8\" (UID: \"259ec8ed-10bd-4fae-8f7f-97409bd4f54b\") " pod="openshift-network-diagnostics/network-check-target-gxtt8"
Jul 12 17:04:17.164034 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:17.164014    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 12 17:04:17.174318 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:17.174293    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 12 17:04:17.185502 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:17.185479    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6sf9\" (UniqueName: \"kubernetes.io/projected/259ec8ed-10bd-4fae-8f7f-97409bd4f54b-kube-api-access-n6sf9\") pod \"network-check-target-gxtt8\" (UID: \"259ec8ed-10bd-4fae-8f7f-97409bd4f54b\") " pod="openshift-network-diagnostics/network-check-target-gxtt8"
Jul 12 17:04:17.288793 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:17.288763    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mkw2b"
Jul 12 17:04:17.297631 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:17.297608    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-wpssr\""
Jul 12 17:04:17.303958 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:17.303936    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-fgjd4\""
Jul 12 17:04:17.305632 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:17.305612    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h442x"
Jul 12 17:04:17.312354 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:17.312298    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gxtt8"
Jul 12 17:04:17.466366 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:17.465581    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-mkw2b"]
Jul 12 17:04:17.469034 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:04:17.469005    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod577e3a93_0560_4099_bddb_dd55563d002d.slice/crio-a02e2da099f3b61caf4443f82fb4de25e72399e7c68d1c3a67ab1d9db1407b19 WatchSource:0}: Error finding container a02e2da099f3b61caf4443f82fb4de25e72399e7c68d1c3a67ab1d9db1407b19: Status 404 returned error can't find the container with id a02e2da099f3b61caf4443f82fb4de25e72399e7c68d1c3a67ab1d9db1407b19
Jul 12 17:04:17.496204 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:17.496176    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-h442x"]
Jul 12 17:04:17.499400 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:04:17.499369    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1d6c71b_d9fb_4f2a_9a81_5e8f0f8a6c13.slice/crio-05f284d297e42ed31f8543ee618f99356a359ac0b6eeadcb0543e4ed9cdcf0a2 WatchSource:0}: Error finding container 05f284d297e42ed31f8543ee618f99356a359ac0b6eeadcb0543e4ed9cdcf0a2: Status 404 returned error can't find the container with id 05f284d297e42ed31f8543ee618f99356a359ac0b6eeadcb0543e4ed9cdcf0a2
Jul 12 17:04:17.516489 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:17.516416    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-gxtt8"]
Jul 12 17:04:17.519249 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:04:17.519222    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod259ec8ed_10bd_4fae_8f7f_97409bd4f54b.slice/crio-7721fec8f11bd1109b293eebb4235663710283c81669527d988656d93e9f874a WatchSource:0}: Error finding container 7721fec8f11bd1109b293eebb4235663710283c81669527d988656d93e9f874a: Status 404 returned error can't find the container with id 7721fec8f11bd1109b293eebb4235663710283c81669527d988656d93e9f874a
Jul 12 17:04:17.787901 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:17.787855    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-gxtt8" event={"ID":"259ec8ed-10bd-4fae-8f7f-97409bd4f54b","Type":"ContainerStarted","Data":"7721fec8f11bd1109b293eebb4235663710283c81669527d988656d93e9f874a"}
Jul 12 17:04:17.789166 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:17.789136    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-mkw2b" event={"ID":"577e3a93-0560-4099-bddb-dd55563d002d","Type":"ContainerStarted","Data":"a02e2da099f3b61caf4443f82fb4de25e72399e7c68d1c3a67ab1d9db1407b19"}
Jul 12 17:04:17.790823 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:17.790801    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-862s7" event={"ID":"dcf4607e-6cb3-4f9c-b3b3-9e2245bc8867","Type":"ContainerStarted","Data":"b8bd21090468490d5489d8963a343a120622200ce9cf7f984619229d986c64a7"}
Jul 12 17:04:17.790992 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:17.790967    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-862s7"
Jul 12 17:04:17.792231 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:17.792204    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-h442x" event={"ID":"f1d6c71b-d9fb-4f2a-9a81-5e8f0f8a6c13","Type":"ContainerStarted","Data":"05f284d297e42ed31f8543ee618f99356a359ac0b6eeadcb0543e4ed9cdcf0a2"}
Jul 12 17:04:17.797110 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:17.797072    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3cec2e0b-d740-4361-b911-222469a7e0a8","Type":"ContainerStarted","Data":"c1f01bf2fc2b3bf02a6d76b70802a14139753854eccb3da336aac645e2b3a2fd"}
Jul 12 17:04:17.797239 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:17.797216    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-862s7"
Jul 12 17:04:17.807649 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:17.807255    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-862s7" podStartSLOduration=3.676287787 podStartE2EDuration="4.807239414s" totalImagesPullingTime="1.130951627s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 17:04:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 17:04:17.807002564 +0000 UTC m=+65.927319362" watchObservedRunningTime="2026-07-12 17:04:17.807239414 +0000 UTC m=+65.927556215"
Jul 12 17:04:17.837354 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:17.837310    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.986441512 podStartE2EDuration="7.837293647s" totalImagesPullingTime="3.827235949s" totalInitContainerRuntime="23.616186ms" isStatefulPod=false podCreationTimestamp="2026-07-12 17:04:10 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 17:04:17.833990922 +0000 UTC m=+65.954307740" watchObservedRunningTime="2026-07-12 17:04:17.837293647 +0000 UTC m=+65.957610577"
Jul 12 17:04:18.807860 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:18.806778    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-h442x" event={"ID":"f1d6c71b-d9fb-4f2a-9a81-5e8f0f8a6c13","Type":"ContainerStarted","Data":"57424730daae4835e308705745c182857b0bf0f1f0730a47177e95964dc7653b"}
Jul 12 17:04:19.161677 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:19.161628    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5ff95b89fb-ccqfl"]
Jul 12 17:04:19.815412 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:19.815369    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-h442x" event={"ID":"f1d6c71b-d9fb-4f2a-9a81-5e8f0f8a6c13","Type":"ContainerStarted","Data":"710abcfeabcee61f0b5d104dc23601aae0413ff1d13d804a928243895812f10c"}
Jul 12 17:04:19.833926 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:19.833686    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-h442x" podStartSLOduration=66.7022937 podStartE2EDuration="1m7.833664067s" totalImagesPullingTime="1.131370367s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 17:03:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 17:04:19.832469049 +0000 UTC m=+67.952785846" watchObservedRunningTime="2026-07-12 17:04:19.833664067 +0000 UTC m=+67.953980866"
Jul 12 17:04:23.836209 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:23.836162    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-gxtt8" event={"ID":"259ec8ed-10bd-4fae-8f7f-97409bd4f54b","Type":"ContainerStarted","Data":"fc87cf2f951f18a16c27fd154ace2bcd3cbf930b01b12b057948df6fc1477c58"}
Jul 12 17:04:23.836608 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:23.836291    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-gxtt8"
Jul 12 17:04:23.838217 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:23.838191    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-mkw2b" event={"ID":"577e3a93-0560-4099-bddb-dd55563d002d","Type":"ContainerStarted","Data":"2996f667851752fe7a0b255a5d0c34dae51dd03728e9dbc45c480e6b5c3ab900"}
Jul 12 17:04:23.840715 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:23.840693    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea","Type":"ContainerStarted","Data":"6b068d2fa8eefc331d6406452cc63366416fec682eb2dba4c04fac9cb9789edf"}
Jul 12 17:04:23.840812 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:23.840758    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea","Type":"ContainerStarted","Data":"ee358c37ed01d72a985ffbc691b2ff16825a541b53661cec266cbeb8e6d66790"}
Jul 12 17:04:23.859159 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:23.859111    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-gxtt8" podStartSLOduration=66.279544698 podStartE2EDuration="1m11.859092837s" totalImagesPullingTime="5.579548139s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 17:03:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 17:04:23.857137519 +0000 UTC m=+71.977454521" watchObservedRunningTime="2026-07-12 17:04:23.859092837 +0000 UTC m=+71.979409637"
Jul 12 17:04:23.872996 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:23.872946    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-mkw2b" podStartSLOduration=66.23892426 podStartE2EDuration="1m11.87293158s" totalImagesPullingTime="5.63400732s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-12 17:03:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 17:04:23.871827238 +0000 UTC m=+71.992144036" watchObservedRunningTime="2026-07-12 17:04:23.87293158 +0000 UTC m=+71.993248377"
Jul 12 17:04:24.097626 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:24.097546    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5ff95b89fb-ccqfl"
Jul 12 17:04:24.849513 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:24.849483    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea","Type":"ContainerStarted","Data":"be6db67c15c90939cbcd0808881e2846c2bf6da6ff54086809513d1f67690f91"}
Jul 12 17:04:25.856322 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:25.856289    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea","Type":"ContainerStarted","Data":"e197714d1dac18f9ca2021d8560c716caad667fea6320cc4580bcfb46655b7d0"}
Jul 12 17:04:25.856774 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:25.856349    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea","Type":"ContainerStarted","Data":"cab5febf942dd06a1f64f68038217812d5c907d2e346aef30bd2b6608a80c8b6"}
Jul 12 17:04:25.856774 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:25.856376    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea","Type":"ContainerStarted","Data":"f6c03ddd3e3c7cd78a99ee6800bdddac49d6e96fafa45191d36cd152b958012d"}
Jul 12 17:04:25.908673 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:25.908617    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.055182359 podStartE2EDuration="10.908596377s" totalImagesPullingTime="7.819414962s" totalInitContainerRuntime="33.999056ms" isStatefulPod=false podCreationTimestamp="2026-07-12 17:04:15 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 17:04:25.906583189 +0000 UTC m=+74.026899987" watchObservedRunningTime="2026-07-12 17:04:25.908596377 +0000 UTC m=+74.028913178"
Jul 12 17:04:30.493026 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:30.492986    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 12 17:04:30.556227 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:30.556193    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-55b6b74c7-5jbs4"]
Jul 12 17:04:33.520954 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:33.520924    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-5866f48f55-vmnx5"
Jul 12 17:04:33.521321 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:33.521008    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-5866f48f55-vmnx5"
Jul 12 17:04:44.191899 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:44.191751    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5ff95b89fb-ccqfl" podUID="b4d58845-a8df-44ff-ada6-9e794ad37f51" containerName="console" containerID="cri-o://3ccb29edf5f2c04eb590b42afe6949d4a28420b03011a2b20a5a295c5a0e98c0" gracePeriod=15
Jul 12 17:04:44.928813 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:44.928769    2569 generic.go:356] "Generic (PLEG): container finished" podID="b4d58845-a8df-44ff-ada6-9e794ad37f51" containerID="3ccb29edf5f2c04eb590b42afe6949d4a28420b03011a2b20a5a295c5a0e98c0" exitCode=0
Jul 12 17:04:44.928946 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:44.928839    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5ff95b89fb-ccqfl" event={"ID":"b4d58845-a8df-44ff-ada6-9e794ad37f51","Type":"ContainerDied","Data":"3ccb29edf5f2c04eb590b42afe6949d4a28420b03011a2b20a5a295c5a0e98c0"}
Jul 12 17:04:44.928946 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:44.928899    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5ff95b89fb-ccqfl" event={"ID":"b4d58845-a8df-44ff-ada6-9e794ad37f51","Type":"ContainerDied","Data":"7ae1250091df3b811a4c2f42151c093a145b035dbc750f12231c01e4bd473625"}
Jul 12 17:04:44.928946 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:44.928920    2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7ae1250091df3b811a4c2f42151c093a145b035dbc750f12231c01e4bd473625"
Jul 12 17:04:44.935520 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:44.935487    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5ff95b89fb-ccqfl"
Jul 12 17:04:45.095917 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:45.095831    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b4d58845-a8df-44ff-ada6-9e794ad37f51-oauth-serving-cert\") pod \"b4d58845-a8df-44ff-ada6-9e794ad37f51\" (UID: \"b4d58845-a8df-44ff-ada6-9e794ad37f51\") "
Jul 12 17:04:45.095917 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:45.095871    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fqkvh\" (UniqueName: \"kubernetes.io/projected/b4d58845-a8df-44ff-ada6-9e794ad37f51-kube-api-access-fqkvh\") pod \"b4d58845-a8df-44ff-ada6-9e794ad37f51\" (UID: \"b4d58845-a8df-44ff-ada6-9e794ad37f51\") "
Jul 12 17:04:45.095917 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:45.095893    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b4d58845-a8df-44ff-ada6-9e794ad37f51-console-serving-cert\") pod \"b4d58845-a8df-44ff-ada6-9e794ad37f51\" (UID: \"b4d58845-a8df-44ff-ada6-9e794ad37f51\") "
Jul 12 17:04:45.096184 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:45.096014    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b4d58845-a8df-44ff-ada6-9e794ad37f51-service-ca\") pod \"b4d58845-a8df-44ff-ada6-9e794ad37f51\" (UID: \"b4d58845-a8df-44ff-ada6-9e794ad37f51\") "
Jul 12 17:04:45.096184 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:45.096051    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b4d58845-a8df-44ff-ada6-9e794ad37f51-console-config\") pod \"b4d58845-a8df-44ff-ada6-9e794ad37f51\" (UID: \"b4d58845-a8df-44ff-ada6-9e794ad37f51\") "
Jul 12 17:04:45.096184 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:45.096078    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4d58845-a8df-44ff-ada6-9e794ad37f51-trusted-ca-bundle\") pod \"b4d58845-a8df-44ff-ada6-9e794ad37f51\" (UID: \"b4d58845-a8df-44ff-ada6-9e794ad37f51\") "
Jul 12 17:04:45.096184 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:45.096126    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b4d58845-a8df-44ff-ada6-9e794ad37f51-console-oauth-config\") pod \"b4d58845-a8df-44ff-ada6-9e794ad37f51\" (UID: \"b4d58845-a8df-44ff-ada6-9e794ad37f51\") "
Jul 12 17:04:45.096388 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:45.096211    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4d58845-a8df-44ff-ada6-9e794ad37f51-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "b4d58845-a8df-44ff-ada6-9e794ad37f51" (UID: "b4d58845-a8df-44ff-ada6-9e794ad37f51"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 12 17:04:45.096443 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:45.096417    2569 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b4d58845-a8df-44ff-ada6-9e794ad37f51-oauth-serving-cert\") on node \"ip-10-0-141-94.ec2.internal\" DevicePath \"\""
Jul 12 17:04:45.096443 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:45.096413    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4d58845-a8df-44ff-ada6-9e794ad37f51-service-ca" (OuterVolumeSpecName: "service-ca") pod "b4d58845-a8df-44ff-ada6-9e794ad37f51" (UID: "b4d58845-a8df-44ff-ada6-9e794ad37f51"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 12 17:04:45.096579 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:45.096432    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4d58845-a8df-44ff-ada6-9e794ad37f51-console-config" (OuterVolumeSpecName: "console-config") pod "b4d58845-a8df-44ff-ada6-9e794ad37f51" (UID: "b4d58845-a8df-44ff-ada6-9e794ad37f51"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 12 17:04:45.096579 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:45.096550    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4d58845-a8df-44ff-ada6-9e794ad37f51-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "b4d58845-a8df-44ff-ada6-9e794ad37f51" (UID: "b4d58845-a8df-44ff-ada6-9e794ad37f51"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 12 17:04:45.098251 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:45.098216    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4d58845-a8df-44ff-ada6-9e794ad37f51-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "b4d58845-a8df-44ff-ada6-9e794ad37f51" (UID: "b4d58845-a8df-44ff-ada6-9e794ad37f51"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 12 17:04:45.098348 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:45.098260    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4d58845-a8df-44ff-ada6-9e794ad37f51-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "b4d58845-a8df-44ff-ada6-9e794ad37f51" (UID: "b4d58845-a8df-44ff-ada6-9e794ad37f51"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 12 17:04:45.098348 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:45.098256    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4d58845-a8df-44ff-ada6-9e794ad37f51-kube-api-access-fqkvh" (OuterVolumeSpecName: "kube-api-access-fqkvh") pod "b4d58845-a8df-44ff-ada6-9e794ad37f51" (UID: "b4d58845-a8df-44ff-ada6-9e794ad37f51"). InnerVolumeSpecName "kube-api-access-fqkvh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 12 17:04:45.197759 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:45.197731    2569 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b4d58845-a8df-44ff-ada6-9e794ad37f51-service-ca\") on node \"ip-10-0-141-94.ec2.internal\" DevicePath \"\""
Jul 12 17:04:45.197759 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:45.197754    2569 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b4d58845-a8df-44ff-ada6-9e794ad37f51-console-config\") on node \"ip-10-0-141-94.ec2.internal\" DevicePath \"\""
Jul 12 17:04:45.197759 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:45.197763    2569 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4d58845-a8df-44ff-ada6-9e794ad37f51-trusted-ca-bundle\") on node \"ip-10-0-141-94.ec2.internal\" DevicePath \"\""
Jul 12 17:04:45.198147 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:45.197772    2569 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b4d58845-a8df-44ff-ada6-9e794ad37f51-console-oauth-config\") on node \"ip-10-0-141-94.ec2.internal\" DevicePath \"\""
Jul 12 17:04:45.198147 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:45.197780    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqkvh\" (UniqueName: \"kubernetes.io/projected/b4d58845-a8df-44ff-ada6-9e794ad37f51-kube-api-access-fqkvh\") on node \"ip-10-0-141-94.ec2.internal\" DevicePath \"\""
Jul 12 17:04:45.198147 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:45.197789    2569 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b4d58845-a8df-44ff-ada6-9e794ad37f51-console-serving-cert\") on node \"ip-10-0-141-94.ec2.internal\" DevicePath \"\""
Jul 12 17:04:45.932152 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:45.932121    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5ff95b89fb-ccqfl"
Jul 12 17:04:45.951322 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:45.951290    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5ff95b89fb-ccqfl"]
Jul 12 17:04:45.955470 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:45.955446    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5ff95b89fb-ccqfl"]
Jul 12 17:04:46.378535 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:46.378450    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b4d58845-a8df-44ff-ada6-9e794ad37f51" path="/var/lib/kubelet/pods/b4d58845-a8df-44ff-ada6-9e794ad37f51/volumes"
Jul 12 17:04:53.526705 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:53.526674    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-5866f48f55-vmnx5"
Jul 12 17:04:53.530467 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:53.530445    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-5866f48f55-vmnx5"
Jul 12 17:04:54.852529 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:54.852484    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-gxtt8"
Jul 12 17:04:55.576045 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:55.576004    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-55b6b74c7-5jbs4" podUID="37783a93-d722-4cb5-b5d3-7e1cb8726990" containerName="console" containerID="cri-o://2678a40659b3c25aa857a7f3677eb25a9573bab7367a5e1704d3209c3c4f147d" gracePeriod=15
Jul 12 17:04:55.841284 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:55.841229    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55b6b74c7-5jbs4"
Jul 12 17:04:55.878984 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:55.878960    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/37783a93-d722-4cb5-b5d3-7e1cb8726990-console-serving-cert\") pod \"37783a93-d722-4cb5-b5d3-7e1cb8726990\" (UID: \"37783a93-d722-4cb5-b5d3-7e1cb8726990\") "
Jul 12 17:04:55.879309 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:55.878993    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vw8w6\" (UniqueName: \"kubernetes.io/projected/37783a93-d722-4cb5-b5d3-7e1cb8726990-kube-api-access-vw8w6\") pod \"37783a93-d722-4cb5-b5d3-7e1cb8726990\" (UID: \"37783a93-d722-4cb5-b5d3-7e1cb8726990\") "
Jul 12 17:04:55.879309 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:55.879025    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/37783a93-d722-4cb5-b5d3-7e1cb8726990-service-ca\") pod \"37783a93-d722-4cb5-b5d3-7e1cb8726990\" (UID: \"37783a93-d722-4cb5-b5d3-7e1cb8726990\") "
Jul 12 17:04:55.879309 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:55.879071    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/37783a93-d722-4cb5-b5d3-7e1cb8726990-console-config\") pod \"37783a93-d722-4cb5-b5d3-7e1cb8726990\" (UID: \"37783a93-d722-4cb5-b5d3-7e1cb8726990\") "
Jul 12 17:04:55.879309 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:55.879182    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/37783a93-d722-4cb5-b5d3-7e1cb8726990-console-oauth-config\") pod \"37783a93-d722-4cb5-b5d3-7e1cb8726990\" (UID: \"37783a93-d722-4cb5-b5d3-7e1cb8726990\") "
Jul 12 17:04:55.879309 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:55.879249    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/37783a93-d722-4cb5-b5d3-7e1cb8726990-oauth-serving-cert\") pod \"37783a93-d722-4cb5-b5d3-7e1cb8726990\" (UID: \"37783a93-d722-4cb5-b5d3-7e1cb8726990\") "
Jul 12 17:04:55.879578 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:55.879396    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37783a93-d722-4cb5-b5d3-7e1cb8726990-console-config" (OuterVolumeSpecName: "console-config") pod "37783a93-d722-4cb5-b5d3-7e1cb8726990" (UID: "37783a93-d722-4cb5-b5d3-7e1cb8726990"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 12 17:04:55.879578 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:55.879392    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37783a93-d722-4cb5-b5d3-7e1cb8726990-service-ca" (OuterVolumeSpecName: "service-ca") pod "37783a93-d722-4cb5-b5d3-7e1cb8726990" (UID: "37783a93-d722-4cb5-b5d3-7e1cb8726990"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 12 17:04:55.879578 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:55.879469    2569 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/37783a93-d722-4cb5-b5d3-7e1cb8726990-console-config\") on node \"ip-10-0-141-94.ec2.internal\" DevicePath \"\""
Jul 12 17:04:55.879578 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:55.879489    2569 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/37783a93-d722-4cb5-b5d3-7e1cb8726990-service-ca\") on node \"ip-10-0-141-94.ec2.internal\" DevicePath \"\""
Jul 12 17:04:55.879789 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:55.879743    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37783a93-d722-4cb5-b5d3-7e1cb8726990-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "37783a93-d722-4cb5-b5d3-7e1cb8726990" (UID: "37783a93-d722-4cb5-b5d3-7e1cb8726990"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 12 17:04:55.881161 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:55.881133    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37783a93-d722-4cb5-b5d3-7e1cb8726990-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "37783a93-d722-4cb5-b5d3-7e1cb8726990" (UID: "37783a93-d722-4cb5-b5d3-7e1cb8726990"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 12 17:04:55.881236 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:55.881206    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37783a93-d722-4cb5-b5d3-7e1cb8726990-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "37783a93-d722-4cb5-b5d3-7e1cb8726990" (UID: "37783a93-d722-4cb5-b5d3-7e1cb8726990"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 12 17:04:55.881274 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:55.881205    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37783a93-d722-4cb5-b5d3-7e1cb8726990-kube-api-access-vw8w6" (OuterVolumeSpecName: "kube-api-access-vw8w6") pod "37783a93-d722-4cb5-b5d3-7e1cb8726990" (UID: "37783a93-d722-4cb5-b5d3-7e1cb8726990"). InnerVolumeSpecName "kube-api-access-vw8w6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 12 17:04:55.969263 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:55.969232    2569 generic.go:356] "Generic (PLEG): container finished" podID="37783a93-d722-4cb5-b5d3-7e1cb8726990" containerID="2678a40659b3c25aa857a7f3677eb25a9573bab7367a5e1704d3209c3c4f147d" exitCode=0
Jul 12 17:04:55.969395 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:55.969279    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55b6b74c7-5jbs4"
Jul 12 17:04:55.969395 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:55.969321    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55b6b74c7-5jbs4" event={"ID":"37783a93-d722-4cb5-b5d3-7e1cb8726990","Type":"ContainerDied","Data":"2678a40659b3c25aa857a7f3677eb25a9573bab7367a5e1704d3209c3c4f147d"}
Jul 12 17:04:55.969395 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:55.969375    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55b6b74c7-5jbs4" event={"ID":"37783a93-d722-4cb5-b5d3-7e1cb8726990","Type":"ContainerDied","Data":"a460a27b17a0480a5c06215b6aeae1080bc6e6aee0bda0fbfde0d2b63c8873be"}
Jul 12 17:04:55.969528 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:55.969401    2569 scope.go:117] "RemoveContainer" containerID="2678a40659b3c25aa857a7f3677eb25a9573bab7367a5e1704d3209c3c4f147d"
Jul 12 17:04:55.980013 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:55.979992    2569 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/37783a93-d722-4cb5-b5d3-7e1cb8726990-oauth-serving-cert\") on node \"ip-10-0-141-94.ec2.internal\" DevicePath \"\""
Jul 12 17:04:55.980107 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:55.980018    2569 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/37783a93-d722-4cb5-b5d3-7e1cb8726990-console-serving-cert\") on node \"ip-10-0-141-94.ec2.internal\" DevicePath \"\""
Jul 12 17:04:55.980107 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:55.980034    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vw8w6\" (UniqueName: \"kubernetes.io/projected/37783a93-d722-4cb5-b5d3-7e1cb8726990-kube-api-access-vw8w6\") on node \"ip-10-0-141-94.ec2.internal\" DevicePath \"\""
Jul 12 17:04:55.980107 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:55.980048    2569 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/37783a93-d722-4cb5-b5d3-7e1cb8726990-console-oauth-config\") on node \"ip-10-0-141-94.ec2.internal\" DevicePath \"\""
Jul 12 17:04:55.983760 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:55.983739    2569 scope.go:117] "RemoveContainer" containerID="2678a40659b3c25aa857a7f3677eb25a9573bab7367a5e1704d3209c3c4f147d"
Jul 12 17:04:55.984014 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:04:55.983994    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2678a40659b3c25aa857a7f3677eb25a9573bab7367a5e1704d3209c3c4f147d\": container with ID starting with 2678a40659b3c25aa857a7f3677eb25a9573bab7367a5e1704d3209c3c4f147d not found: ID does not exist" containerID="2678a40659b3c25aa857a7f3677eb25a9573bab7367a5e1704d3209c3c4f147d"
Jul 12 17:04:55.984061 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:55.984022    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2678a40659b3c25aa857a7f3677eb25a9573bab7367a5e1704d3209c3c4f147d"} err="failed to get container status \"2678a40659b3c25aa857a7f3677eb25a9573bab7367a5e1704d3209c3c4f147d\": rpc error: code = NotFound desc = could not find container \"2678a40659b3c25aa857a7f3677eb25a9573bab7367a5e1704d3209c3c4f147d\": container with ID starting with 2678a40659b3c25aa857a7f3677eb25a9573bab7367a5e1704d3209c3c4f147d not found: ID does not exist"
Jul 12 17:04:55.988995 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:55.988976    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-55b6b74c7-5jbs4"]
Jul 12 17:04:55.993280 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:55.993262    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-55b6b74c7-5jbs4"]
Jul 12 17:04:56.378498 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:04:56.378471    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="37783a93-d722-4cb5-b5d3-7e1cb8726990" path="/var/lib/kubelet/pods/37783a93-d722-4cb5-b5d3-7e1cb8726990/volumes"
Jul 12 17:05:15.493425 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:15.493375    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 12 17:05:15.512991 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:15.512966    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 12 17:05:16.063971 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:16.063935    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 12 17:05:29.330895 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:29.330858    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 12 17:05:29.331306 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:29.331168    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="3cec2e0b-d740-4361-b911-222469a7e0a8" containerName="alertmanager" containerID="cri-o://ee9b58944010e534f99a91c97f2febe666a9b23d85a88877f476b77c77bbca35" gracePeriod=120
Jul 12 17:05:29.331306 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:29.331201    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="3cec2e0b-d740-4361-b911-222469a7e0a8" containerName="kube-rbac-proxy-metric" containerID="cri-o://9a127ff898b7d40306157481a7e12303476ac05ef2d4cd1671e126cf4747ed93" gracePeriod=120
Jul 12 17:05:29.331306 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:29.331248    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="3cec2e0b-d740-4361-b911-222469a7e0a8" containerName="config-reloader" containerID="cri-o://f8ea0483a2b54598b42de1bd70aa1a8e0c2f6bc13c7a360099b51306ac3fd594" gracePeriod=120
Jul 12 17:05:29.331306 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:29.331249    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="3cec2e0b-d740-4361-b911-222469a7e0a8" containerName="kube-rbac-proxy-web" containerID="cri-o://51dd575b946a722aa4bd01f79c91647b4c3bdfaabf6ea3248ac22695b653933f" gracePeriod=120
Jul 12 17:05:29.331562 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:29.331286    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="3cec2e0b-d740-4361-b911-222469a7e0a8" containerName="kube-rbac-proxy" containerID="cri-o://84664ee66a8f4c2fe0824b1c2a99440723a48f9c8d6e0e06ba71c731c973f0e3" gracePeriod=120
Jul 12 17:05:29.331562 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:29.331267    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="3cec2e0b-d740-4361-b911-222469a7e0a8" containerName="prom-label-proxy" containerID="cri-o://c1f01bf2fc2b3bf02a6d76b70802a14139753854eccb3da336aac645e2b3a2fd" gracePeriod=120
Jul 12 17:05:30.106143 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:30.106105    2569 generic.go:356] "Generic (PLEG): container finished" podID="3cec2e0b-d740-4361-b911-222469a7e0a8" containerID="c1f01bf2fc2b3bf02a6d76b70802a14139753854eccb3da336aac645e2b3a2fd" exitCode=0
Jul 12 17:05:30.106143 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:30.106134    2569 generic.go:356] "Generic (PLEG): container finished" podID="3cec2e0b-d740-4361-b911-222469a7e0a8" containerID="9a127ff898b7d40306157481a7e12303476ac05ef2d4cd1671e126cf4747ed93" exitCode=0
Jul 12 17:05:30.106143 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:30.106140    2569 generic.go:356] "Generic (PLEG): container finished" podID="3cec2e0b-d740-4361-b911-222469a7e0a8" containerID="84664ee66a8f4c2fe0824b1c2a99440723a48f9c8d6e0e06ba71c731c973f0e3" exitCode=0
Jul 12 17:05:30.106143 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:30.106145    2569 generic.go:356] "Generic (PLEG): container finished" podID="3cec2e0b-d740-4361-b911-222469a7e0a8" containerID="f8ea0483a2b54598b42de1bd70aa1a8e0c2f6bc13c7a360099b51306ac3fd594" exitCode=0
Jul 12 17:05:30.106143 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:30.106150    2569 generic.go:356] "Generic (PLEG): container finished" podID="3cec2e0b-d740-4361-b911-222469a7e0a8" containerID="ee9b58944010e534f99a91c97f2febe666a9b23d85a88877f476b77c77bbca35" exitCode=0
Jul 12 17:05:30.106427 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:30.106177    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3cec2e0b-d740-4361-b911-222469a7e0a8","Type":"ContainerDied","Data":"c1f01bf2fc2b3bf02a6d76b70802a14139753854eccb3da336aac645e2b3a2fd"}
Jul 12 17:05:30.106427 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:30.106264    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3cec2e0b-d740-4361-b911-222469a7e0a8","Type":"ContainerDied","Data":"9a127ff898b7d40306157481a7e12303476ac05ef2d4cd1671e126cf4747ed93"}
Jul 12 17:05:30.106427 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:30.106291    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3cec2e0b-d740-4361-b911-222469a7e0a8","Type":"ContainerDied","Data":"84664ee66a8f4c2fe0824b1c2a99440723a48f9c8d6e0e06ba71c731c973f0e3"}
Jul 12 17:05:30.106427 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:30.106320    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3cec2e0b-d740-4361-b911-222469a7e0a8","Type":"ContainerDied","Data":"f8ea0483a2b54598b42de1bd70aa1a8e0c2f6bc13c7a360099b51306ac3fd594"}
Jul 12 17:05:30.106427 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:30.106344    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3cec2e0b-d740-4361-b911-222469a7e0a8","Type":"ContainerDied","Data":"ee9b58944010e534f99a91c97f2febe666a9b23d85a88877f476b77c77bbca35"}
Jul 12 17:05:30.661935 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:30.661913    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 12 17:05:30.756233 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:30.756157    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3cec2e0b-d740-4361-b911-222469a7e0a8-config-out\") pod \"3cec2e0b-d740-4361-b911-222469a7e0a8\" (UID: \"3cec2e0b-d740-4361-b911-222469a7e0a8\") "
Jul 12 17:05:30.756233 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:30.756192    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3cec2e0b-d740-4361-b911-222469a7e0a8-web-config\") pod \"3cec2e0b-d740-4361-b911-222469a7e0a8\" (UID: \"3cec2e0b-d740-4361-b911-222469a7e0a8\") "
Jul 12 17:05:30.756233 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:30.756212    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3cec2e0b-d740-4361-b911-222469a7e0a8-secret-alertmanager-main-tls\") pod \"3cec2e0b-d740-4361-b911-222469a7e0a8\" (UID: \"3cec2e0b-d740-4361-b911-222469a7e0a8\") "
Jul 12 17:05:30.756491 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:30.756262    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/3cec2e0b-d740-4361-b911-222469a7e0a8-secret-alertmanager-kube-rbac-proxy-metric\") pod \"3cec2e0b-d740-4361-b911-222469a7e0a8\" (UID: \"3cec2e0b-d740-4361-b911-222469a7e0a8\") "
Jul 12 17:05:30.756491 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:30.756280    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3cec2e0b-d740-4361-b911-222469a7e0a8-secret-alertmanager-kube-rbac-proxy\") pod \"3cec2e0b-d740-4361-b911-222469a7e0a8\" (UID: \"3cec2e0b-d740-4361-b911-222469a7e0a8\") "
Jul 12 17:05:30.756491 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:30.756303    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3cec2e0b-d740-4361-b911-222469a7e0a8-alertmanager-trusted-ca-bundle\") pod \"3cec2e0b-d740-4361-b911-222469a7e0a8\" (UID: \"3cec2e0b-d740-4361-b911-222469a7e0a8\") "
Jul 12 17:05:30.756491 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:30.756319    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3cec2e0b-d740-4361-b911-222469a7e0a8-config-volume\") pod \"3cec2e0b-d740-4361-b911-222469a7e0a8\" (UID: \"3cec2e0b-d740-4361-b911-222469a7e0a8\") "
Jul 12 17:05:30.756491 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:30.756343    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3cec2e0b-d740-4361-b911-222469a7e0a8-tls-assets\") pod \"3cec2e0b-d740-4361-b911-222469a7e0a8\" (UID: \"3cec2e0b-d740-4361-b911-222469a7e0a8\") "
Jul 12 17:05:30.756491 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:30.756373    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sw92s\" (UniqueName: \"kubernetes.io/projected/3cec2e0b-d740-4361-b911-222469a7e0a8-kube-api-access-sw92s\") pod \"3cec2e0b-d740-4361-b911-222469a7e0a8\" (UID: \"3cec2e0b-d740-4361-b911-222469a7e0a8\") "
Jul 12 17:05:30.756491 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:30.756406    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/3cec2e0b-d740-4361-b911-222469a7e0a8-alertmanager-main-db\") pod \"3cec2e0b-d740-4361-b911-222469a7e0a8\" (UID: \"3cec2e0b-d740-4361-b911-222469a7e0a8\") "
Jul 12 17:05:30.756491 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:30.756434    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3cec2e0b-d740-4361-b911-222469a7e0a8-secret-alertmanager-kube-rbac-proxy-web\") pod \"3cec2e0b-d740-4361-b911-222469a7e0a8\" (UID: \"3cec2e0b-d740-4361-b911-222469a7e0a8\") "
Jul 12 17:05:30.756491 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:30.756458    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3cec2e0b-d740-4361-b911-222469a7e0a8-metrics-client-ca\") pod \"3cec2e0b-d740-4361-b911-222469a7e0a8\" (UID: \"3cec2e0b-d740-4361-b911-222469a7e0a8\") "
Jul 12 17:05:30.756957 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:30.756502    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3cec2e0b-d740-4361-b911-222469a7e0a8-cluster-tls-config\") pod \"3cec2e0b-d740-4361-b911-222469a7e0a8\" (UID: \"3cec2e0b-d740-4361-b911-222469a7e0a8\") "
Jul 12 17:05:30.756957 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:30.756745    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cec2e0b-d740-4361-b911-222469a7e0a8-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "3cec2e0b-d740-4361-b911-222469a7e0a8" (UID: "3cec2e0b-d740-4361-b911-222469a7e0a8"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 12 17:05:30.757104 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:30.757078    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cec2e0b-d740-4361-b911-222469a7e0a8-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "3cec2e0b-d740-4361-b911-222469a7e0a8" (UID: "3cec2e0b-d740-4361-b911-222469a7e0a8"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 12 17:05:30.758473 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:30.758447    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cec2e0b-d740-4361-b911-222469a7e0a8-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "3cec2e0b-d740-4361-b911-222469a7e0a8" (UID: "3cec2e0b-d740-4361-b911-222469a7e0a8"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 12 17:05:30.759396 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:30.759371    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cec2e0b-d740-4361-b911-222469a7e0a8-config-out" (OuterVolumeSpecName: "config-out") pod "3cec2e0b-d740-4361-b911-222469a7e0a8" (UID: "3cec2e0b-d740-4361-b911-222469a7e0a8"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 12 17:05:30.759489 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:30.759463    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3cec2e0b-d740-4361-b911-222469a7e0a8-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "3cec2e0b-d740-4361-b911-222469a7e0a8" (UID: "3cec2e0b-d740-4361-b911-222469a7e0a8"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 12 17:05:30.760018 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:30.759908    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3cec2e0b-d740-4361-b911-222469a7e0a8-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "3cec2e0b-d740-4361-b911-222469a7e0a8" (UID: "3cec2e0b-d740-4361-b911-222469a7e0a8"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 12 17:05:30.760018 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:30.760002    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cec2e0b-d740-4361-b911-222469a7e0a8-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "3cec2e0b-d740-4361-b911-222469a7e0a8" (UID: "3cec2e0b-d740-4361-b911-222469a7e0a8"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 12 17:05:30.760183 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:30.760111    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3cec2e0b-d740-4361-b911-222469a7e0a8-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "3cec2e0b-d740-4361-b911-222469a7e0a8" (UID: "3cec2e0b-d740-4361-b911-222469a7e0a8"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 12 17:05:30.760528 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:30.760482    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3cec2e0b-d740-4361-b911-222469a7e0a8-config-volume" (OuterVolumeSpecName: "config-volume") pod "3cec2e0b-d740-4361-b911-222469a7e0a8" (UID: "3cec2e0b-d740-4361-b911-222469a7e0a8"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 12 17:05:30.760610 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:30.760588    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cec2e0b-d740-4361-b911-222469a7e0a8-kube-api-access-sw92s" (OuterVolumeSpecName: "kube-api-access-sw92s") pod "3cec2e0b-d740-4361-b911-222469a7e0a8" (UID: "3cec2e0b-d740-4361-b911-222469a7e0a8"). InnerVolumeSpecName "kube-api-access-sw92s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 12 17:05:30.761267 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:30.761242    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3cec2e0b-d740-4361-b911-222469a7e0a8-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "3cec2e0b-d740-4361-b911-222469a7e0a8" (UID: "3cec2e0b-d740-4361-b911-222469a7e0a8"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 12 17:05:30.763710 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:30.763689    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3cec2e0b-d740-4361-b911-222469a7e0a8-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "3cec2e0b-d740-4361-b911-222469a7e0a8" (UID: "3cec2e0b-d740-4361-b911-222469a7e0a8"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 12 17:05:30.769407 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:30.769382    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3cec2e0b-d740-4361-b911-222469a7e0a8-web-config" (OuterVolumeSpecName: "web-config") pod "3cec2e0b-d740-4361-b911-222469a7e0a8" (UID: "3cec2e0b-d740-4361-b911-222469a7e0a8"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 12 17:05:30.857857 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:30.857835    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sw92s\" (UniqueName: \"kubernetes.io/projected/3cec2e0b-d740-4361-b911-222469a7e0a8-kube-api-access-sw92s\") on node \"ip-10-0-141-94.ec2.internal\" DevicePath \"\""
Jul 12 17:05:30.857857 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:30.857856    2569 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/3cec2e0b-d740-4361-b911-222469a7e0a8-alertmanager-main-db\") on node \"ip-10-0-141-94.ec2.internal\" DevicePath \"\""
Jul 12 17:05:30.857968 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:30.857866    2569 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3cec2e0b-d740-4361-b911-222469a7e0a8-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-141-94.ec2.internal\" DevicePath \"\""
Jul 12 17:05:30.857968 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:30.857875    2569 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3cec2e0b-d740-4361-b911-222469a7e0a8-metrics-client-ca\") on node \"ip-10-0-141-94.ec2.internal\" DevicePath \"\""
Jul 12 17:05:30.857968 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:30.857883    2569 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3cec2e0b-d740-4361-b911-222469a7e0a8-cluster-tls-config\") on node \"ip-10-0-141-94.ec2.internal\" DevicePath \"\""
Jul 12 17:05:30.857968 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:30.857892    2569 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3cec2e0b-d740-4361-b911-222469a7e0a8-config-out\") on node \"ip-10-0-141-94.ec2.internal\" DevicePath \"\""
Jul 12 17:05:30.857968 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:30.857899    2569 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3cec2e0b-d740-4361-b911-222469a7e0a8-web-config\") on node \"ip-10-0-141-94.ec2.internal\" DevicePath \"\""
Jul 12 17:05:30.857968 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:30.857907    2569 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3cec2e0b-d740-4361-b911-222469a7e0a8-secret-alertmanager-main-tls\") on node \"ip-10-0-141-94.ec2.internal\" DevicePath \"\""
Jul 12 17:05:30.857968 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:30.857917    2569 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/3cec2e0b-d740-4361-b911-222469a7e0a8-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-141-94.ec2.internal\" DevicePath \"\""
Jul 12 17:05:30.857968 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:30.857925    2569 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3cec2e0b-d740-4361-b911-222469a7e0a8-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-141-94.ec2.internal\" DevicePath \"\""
Jul 12 17:05:30.857968 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:30.857934    2569 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3cec2e0b-d740-4361-b911-222469a7e0a8-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-141-94.ec2.internal\" DevicePath \"\""
Jul 12 17:05:30.857968 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:30.857942    2569 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3cec2e0b-d740-4361-b911-222469a7e0a8-config-volume\") on node \"ip-10-0-141-94.ec2.internal\" DevicePath \"\""
Jul 12 17:05:30.857968 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:30.857949    2569 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3cec2e0b-d740-4361-b911-222469a7e0a8-tls-assets\") on node \"ip-10-0-141-94.ec2.internal\" DevicePath \"\""
Jul 12 17:05:31.112930 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:31.112847    2569 generic.go:356] "Generic (PLEG): container finished" podID="3cec2e0b-d740-4361-b911-222469a7e0a8" containerID="51dd575b946a722aa4bd01f79c91647b4c3bdfaabf6ea3248ac22695b653933f" exitCode=0
Jul 12 17:05:31.112930 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:31.112924    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3cec2e0b-d740-4361-b911-222469a7e0a8","Type":"ContainerDied","Data":"51dd575b946a722aa4bd01f79c91647b4c3bdfaabf6ea3248ac22695b653933f"}
Jul 12 17:05:31.113100 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:31.112933    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 12 17:05:31.113100 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:31.112994    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3cec2e0b-d740-4361-b911-222469a7e0a8","Type":"ContainerDied","Data":"3b92a00d3355f6226378f04146b995c82023e5797f883e6489b9dee5422cb9b1"}
Jul 12 17:05:31.113100 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:31.113029    2569 scope.go:117] "RemoveContainer" containerID="c1f01bf2fc2b3bf02a6d76b70802a14139753854eccb3da336aac645e2b3a2fd"
Jul 12 17:05:31.120657 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:31.120634    2569 scope.go:117] "RemoveContainer" containerID="9a127ff898b7d40306157481a7e12303476ac05ef2d4cd1671e126cf4747ed93"
Jul 12 17:05:31.127706 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:31.127688    2569 scope.go:117] "RemoveContainer" containerID="84664ee66a8f4c2fe0824b1c2a99440723a48f9c8d6e0e06ba71c731c973f0e3"
Jul 12 17:05:31.134014 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:31.133996    2569 scope.go:117] "RemoveContainer" containerID="51dd575b946a722aa4bd01f79c91647b4c3bdfaabf6ea3248ac22695b653933f"
Jul 12 17:05:31.136717 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:31.136695    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 12 17:05:31.141610 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:31.141586    2569 scope.go:117] "RemoveContainer" containerID="f8ea0483a2b54598b42de1bd70aa1a8e0c2f6bc13c7a360099b51306ac3fd594"
Jul 12 17:05:31.143149 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:31.143130    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 12 17:05:31.148339 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:31.148323    2569 scope.go:117] "RemoveContainer" containerID="ee9b58944010e534f99a91c97f2febe666a9b23d85a88877f476b77c77bbca35"
Jul 12 17:05:31.155045 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:31.155031    2569 scope.go:117] "RemoveContainer" containerID="718f703a53ceff3baa84fe3823cdd2cfb380983827b9dd70bdae0cb7c205c364"
Jul 12 17:05:31.161677 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:31.161661    2569 scope.go:117] "RemoveContainer" containerID="c1f01bf2fc2b3bf02a6d76b70802a14139753854eccb3da336aac645e2b3a2fd"
Jul 12 17:05:31.161926 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:05:31.161907    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1f01bf2fc2b3bf02a6d76b70802a14139753854eccb3da336aac645e2b3a2fd\": container with ID starting with c1f01bf2fc2b3bf02a6d76b70802a14139753854eccb3da336aac645e2b3a2fd not found: ID does not exist" containerID="c1f01bf2fc2b3bf02a6d76b70802a14139753854eccb3da336aac645e2b3a2fd"
Jul 12 17:05:31.161966 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:31.161935    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1f01bf2fc2b3bf02a6d76b70802a14139753854eccb3da336aac645e2b3a2fd"} err="failed to get container status \"c1f01bf2fc2b3bf02a6d76b70802a14139753854eccb3da336aac645e2b3a2fd\": rpc error: code = NotFound desc = could not find container \"c1f01bf2fc2b3bf02a6d76b70802a14139753854eccb3da336aac645e2b3a2fd\": container with ID starting with c1f01bf2fc2b3bf02a6d76b70802a14139753854eccb3da336aac645e2b3a2fd not found: ID does not exist"
Jul 12 17:05:31.161966 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:31.161953    2569 scope.go:117] "RemoveContainer" containerID="9a127ff898b7d40306157481a7e12303476ac05ef2d4cd1671e126cf4747ed93"
Jul 12 17:05:31.162164 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:05:31.162147    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a127ff898b7d40306157481a7e12303476ac05ef2d4cd1671e126cf4747ed93\": container with ID starting with 9a127ff898b7d40306157481a7e12303476ac05ef2d4cd1671e126cf4747ed93 not found: ID does not exist" containerID="9a127ff898b7d40306157481a7e12303476ac05ef2d4cd1671e126cf4747ed93"
Jul 12 17:05:31.162204 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:31.162170    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a127ff898b7d40306157481a7e12303476ac05ef2d4cd1671e126cf4747ed93"} err="failed to get container status \"9a127ff898b7d40306157481a7e12303476ac05ef2d4cd1671e126cf4747ed93\": rpc error: code = NotFound desc = could not find container \"9a127ff898b7d40306157481a7e12303476ac05ef2d4cd1671e126cf4747ed93\": container with ID starting with 9a127ff898b7d40306157481a7e12303476ac05ef2d4cd1671e126cf4747ed93 not found: ID does not exist"
Jul 12 17:05:31.162204 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:31.162186    2569 scope.go:117] "RemoveContainer" containerID="84664ee66a8f4c2fe0824b1c2a99440723a48f9c8d6e0e06ba71c731c973f0e3"
Jul 12 17:05:31.162393 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:05:31.162377    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84664ee66a8f4c2fe0824b1c2a99440723a48f9c8d6e0e06ba71c731c973f0e3\": container with ID starting with 84664ee66a8f4c2fe0824b1c2a99440723a48f9c8d6e0e06ba71c731c973f0e3 not found: ID does not exist" containerID="84664ee66a8f4c2fe0824b1c2a99440723a48f9c8d6e0e06ba71c731c973f0e3"
Jul 12 17:05:31.162439 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:31.162398    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84664ee66a8f4c2fe0824b1c2a99440723a48f9c8d6e0e06ba71c731c973f0e3"} err="failed to get container status \"84664ee66a8f4c2fe0824b1c2a99440723a48f9c8d6e0e06ba71c731c973f0e3\": rpc error: code = NotFound desc = could not find container \"84664ee66a8f4c2fe0824b1c2a99440723a48f9c8d6e0e06ba71c731c973f0e3\": container with ID starting with 84664ee66a8f4c2fe0824b1c2a99440723a48f9c8d6e0e06ba71c731c973f0e3 not found: ID does not exist"
Jul 12 17:05:31.162439 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:31.162410    2569 scope.go:117] "RemoveContainer" containerID="51dd575b946a722aa4bd01f79c91647b4c3bdfaabf6ea3248ac22695b653933f"
Jul 12 17:05:31.162647 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:05:31.162631    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51dd575b946a722aa4bd01f79c91647b4c3bdfaabf6ea3248ac22695b653933f\": container with ID starting with 51dd575b946a722aa4bd01f79c91647b4c3bdfaabf6ea3248ac22695b653933f not found: ID does not exist" containerID="51dd575b946a722aa4bd01f79c91647b4c3bdfaabf6ea3248ac22695b653933f"
Jul 12 17:05:31.162708 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:31.162650    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51dd575b946a722aa4bd01f79c91647b4c3bdfaabf6ea3248ac22695b653933f"} err="failed to get container status \"51dd575b946a722aa4bd01f79c91647b4c3bdfaabf6ea3248ac22695b653933f\": rpc error: code = NotFound desc = could not find container \"51dd575b946a722aa4bd01f79c91647b4c3bdfaabf6ea3248ac22695b653933f\": container with ID starting with 51dd575b946a722aa4bd01f79c91647b4c3bdfaabf6ea3248ac22695b653933f not found: ID does not exist"
Jul 12 17:05:31.162708 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:31.162663    2569 scope.go:117] "RemoveContainer" containerID="f8ea0483a2b54598b42de1bd70aa1a8e0c2f6bc13c7a360099b51306ac3fd594"
Jul 12 17:05:31.162924 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:05:31.162909    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8ea0483a2b54598b42de1bd70aa1a8e0c2f6bc13c7a360099b51306ac3fd594\": container with ID starting with f8ea0483a2b54598b42de1bd70aa1a8e0c2f6bc13c7a360099b51306ac3fd594 not found: ID does not exist" containerID="f8ea0483a2b54598b42de1bd70aa1a8e0c2f6bc13c7a360099b51306ac3fd594"
Jul 12 17:05:31.162971 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:31.162928    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8ea0483a2b54598b42de1bd70aa1a8e0c2f6bc13c7a360099b51306ac3fd594"} err="failed to get container status \"f8ea0483a2b54598b42de1bd70aa1a8e0c2f6bc13c7a360099b51306ac3fd594\": rpc error: code = NotFound desc = could not find container \"f8ea0483a2b54598b42de1bd70aa1a8e0c2f6bc13c7a360099b51306ac3fd594\": container with ID starting with f8ea0483a2b54598b42de1bd70aa1a8e0c2f6bc13c7a360099b51306ac3fd594 not found: ID does not exist"
Jul 12 17:05:31.162971 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:31.162940    2569 scope.go:117] "RemoveContainer" containerID="ee9b58944010e534f99a91c97f2febe666a9b23d85a88877f476b77c77bbca35"
Jul 12 17:05:31.163166 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:05:31.163149    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee9b58944010e534f99a91c97f2febe666a9b23d85a88877f476b77c77bbca35\": container with ID starting with ee9b58944010e534f99a91c97f2febe666a9b23d85a88877f476b77c77bbca35 not found: ID does not exist" containerID="ee9b58944010e534f99a91c97f2febe666a9b23d85a88877f476b77c77bbca35"
Jul 12 17:05:31.163206 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:31.163171    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee9b58944010e534f99a91c97f2febe666a9b23d85a88877f476b77c77bbca35"} err="failed to get container status \"ee9b58944010e534f99a91c97f2febe666a9b23d85a88877f476b77c77bbca35\": rpc error: code = NotFound desc = could not find container \"ee9b58944010e534f99a91c97f2febe666a9b23d85a88877f476b77c77bbca35\": container with ID starting with ee9b58944010e534f99a91c97f2febe666a9b23d85a88877f476b77c77bbca35 not found: ID does not exist"
Jul 12 17:05:31.163206 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:31.163190    2569 scope.go:117] "RemoveContainer" containerID="718f703a53ceff3baa84fe3823cdd2cfb380983827b9dd70bdae0cb7c205c364"
Jul 12 17:05:31.163426 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:05:31.163410    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"718f703a53ceff3baa84fe3823cdd2cfb380983827b9dd70bdae0cb7c205c364\": container with ID starting with 718f703a53ceff3baa84fe3823cdd2cfb380983827b9dd70bdae0cb7c205c364 not found: ID does not exist" containerID="718f703a53ceff3baa84fe3823cdd2cfb380983827b9dd70bdae0cb7c205c364"
Jul 12 17:05:31.163464 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:31.163429    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"718f703a53ceff3baa84fe3823cdd2cfb380983827b9dd70bdae0cb7c205c364"} err="failed to get container status \"718f703a53ceff3baa84fe3823cdd2cfb380983827b9dd70bdae0cb7c205c364\": rpc error: code = NotFound desc = could not find container \"718f703a53ceff3baa84fe3823cdd2cfb380983827b9dd70bdae0cb7c205c364\": container with ID starting with 718f703a53ceff3baa84fe3823cdd2cfb380983827b9dd70bdae0cb7c205c364 not found: ID does not exist"
Jul 12 17:05:32.378573 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:32.378536    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3cec2e0b-d740-4361-b911-222469a7e0a8" path="/var/lib/kubelet/pods/3cec2e0b-d740-4361-b911-222469a7e0a8/volumes"
Jul 12 17:05:33.546922 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:33.546885    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 12 17:05:33.547432 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:33.547214    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea" containerName="prometheus" containerID="cri-o://ee358c37ed01d72a985ffbc691b2ff16825a541b53661cec266cbeb8e6d66790" gracePeriod=600
Jul 12 17:05:33.547432 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:33.547236    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea" containerName="kube-rbac-proxy" containerID="cri-o://cab5febf942dd06a1f64f68038217812d5c907d2e346aef30bd2b6608a80c8b6" gracePeriod=600
Jul 12 17:05:33.547432 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:33.547259    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea" containerName="kube-rbac-proxy-thanos" containerID="cri-o://e197714d1dac18f9ca2021d8560c716caad667fea6320cc4580bcfb46655b7d0" gracePeriod=600
Jul 12 17:05:33.547432 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:33.547273    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea" containerName="thanos-sidecar" containerID="cri-o://be6db67c15c90939cbcd0808881e2846c2bf6da6ff54086809513d1f67690f91" gracePeriod=600
Jul 12 17:05:33.547432 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:33.547376    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea" containerName="config-reloader" containerID="cri-o://6b068d2fa8eefc331d6406452cc63366416fec682eb2dba4c04fac9cb9789edf" gracePeriod=600
Jul 12 17:05:33.547432 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:33.547386    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea" containerName="kube-rbac-proxy-web" containerID="cri-o://f6c03ddd3e3c7cd78a99ee6800bdddac49d6e96fafa45191d36cd152b958012d" gracePeriod=600
Jul 12 17:05:33.801268 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:33.800928    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 12 17:05:33.880430 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:33.880375    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea-configmap-metrics-client-ca\") pod \"80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea\" (UID: \"80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea\") "
Jul 12 17:05:33.880621 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:33.880458    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea-web-config\") pod \"80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea\" (UID: \"80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea\") "
Jul 12 17:05:33.880621 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:33.880479    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea-prometheus-k8s-db\") pod \"80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea\" (UID: \"80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea\") "
Jul 12 17:05:33.880621 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:33.880500    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea-configmap-kubelet-serving-ca-bundle\") pod \"80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea\" (UID: \"80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea\") "
Jul 12 17:05:33.880621 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:33.880545    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea-tls-assets\") pod \"80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea\" (UID: \"80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea\") "
Jul 12 17:05:33.880621 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:33.880563    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea-config-out\") pod \"80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea\" (UID: \"80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea\") "
Jul 12 17:05:33.880923 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:33.880645    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea-prometheus-k8s-rulefiles-2\") pod \"80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea\" (UID: \"80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea\") "
Jul 12 17:05:33.880923 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:33.880681    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea\" (UID: \"80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea\") "
Jul 12 17:05:33.880923 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:33.880714    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea-thanos-prometheus-http-client-file\") pod \"80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea\" (UID: \"80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea\") "
Jul 12 17:05:33.880923 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:33.880745    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea-secret-metrics-client-certs\") pod \"80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea\" (UID: \"80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea\") "
Jul 12 17:05:33.880923 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:33.880778    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea-prometheus-trusted-ca-bundle\") pod \"80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea\" (UID: \"80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea\") "
Jul 12 17:05:33.880923 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:33.880778    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea" (UID: "80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 12 17:05:33.880923 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:33.880806    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea-config\") pod \"80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea\" (UID: \"80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea\") "
Jul 12 17:05:33.880923 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:33.880836    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea-prometheus-k8s-rulefiles-0\") pod \"80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea\" (UID: \"80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea\") "
Jul 12 17:05:33.880923 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:33.880864    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea-configmap-serving-certs-ca-bundle\") pod \"80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea\" (UID: \"80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea\") "
Jul 12 17:05:33.880923 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:33.880890    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea" (UID: "80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 12 17:05:33.880923 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:33.880902    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea-secret-kube-rbac-proxy\") pod \"80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea\" (UID: \"80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea\") "
Jul 12 17:05:33.881489 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:33.880978    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea-prometheus-k8s-rulefiles-1\") pod \"80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea\" (UID: \"80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea\") "
Jul 12 17:05:33.881489 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:33.881013    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea-secret-prometheus-k8s-tls\") pod \"80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea\" (UID: \"80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea\") "
Jul 12 17:05:33.881489 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:33.881044    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea-secret-grpc-tls\") pod \"80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea\" (UID: \"80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea\") "
Jul 12 17:05:33.881489 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:33.881072    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mxfwl\" (UniqueName: \"kubernetes.io/projected/80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea-kube-api-access-mxfwl\") pod \"80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea\" (UID: \"80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea\") "
Jul 12 17:05:33.881489 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:33.881097    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea\" (UID: \"80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea\") "
Jul 12 17:05:33.881489 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:33.881313    2569 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea-configmap-metrics-client-ca\") on node \"ip-10-0-141-94.ec2.internal\" DevicePath \"\""
Jul 12 17:05:33.881489 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:33.881328    2569 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-141-94.ec2.internal\" DevicePath \"\""
Jul 12 17:05:33.881489 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:33.881478    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea" (UID: "80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 12 17:05:33.882398 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:33.882033    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea" (UID: "80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 12 17:05:33.882398 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:33.882078    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea" (UID: "80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 12 17:05:33.884179 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:33.882640    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea" (UID: "80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 12 17:05:33.884179 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:33.882662    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea" (UID: "80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 12 17:05:33.884179 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:33.883861    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea" (UID: "80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 12 17:05:33.884564 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:33.884537    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea" (UID: "80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 12 17:05:33.884655 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:33.884538    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea" (UID: "80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 12 17:05:33.885041 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:33.885014    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea" (UID: "80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 12 17:05:33.885746 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:33.885721    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea" (UID: "80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 12 17:05:33.885873 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:33.885844    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea" (UID: "80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 12 17:05:33.886087 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:33.885988    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea" (UID: "80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 12 17:05:33.886087 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:33.886028    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea-config-out" (OuterVolumeSpecName: "config-out") pod "80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea" (UID: "80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 12 17:05:33.886087 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:33.886052    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea-config" (OuterVolumeSpecName: "config") pod "80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea" (UID: "80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 12 17:05:33.886315 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:33.886276    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea-kube-api-access-mxfwl" (OuterVolumeSpecName: "kube-api-access-mxfwl") pod "80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea" (UID: "80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea"). InnerVolumeSpecName "kube-api-access-mxfwl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 12 17:05:33.886847 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:33.886822    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea" (UID: "80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 12 17:05:33.887232 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:33.887214    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea" (UID: "80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 12 17:05:33.894964 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:33.894945    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea-web-config" (OuterVolumeSpecName: "web-config") pod "80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea" (UID: "80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 12 17:05:33.982477 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:33.982443    2569 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea-config\") on node \"ip-10-0-141-94.ec2.internal\" DevicePath \"\""
Jul 12 17:05:33.982477 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:33.982473    2569 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-141-94.ec2.internal\" DevicePath \"\""
Jul 12 17:05:33.982477 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:33.982484    2569 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-141-94.ec2.internal\" DevicePath \"\""
Jul 12 17:05:33.982716 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:33.982493    2569 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea-secret-kube-rbac-proxy\") on node \"ip-10-0-141-94.ec2.internal\" DevicePath \"\""
Jul 12 17:05:33.982716 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:33.982503    2569 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-141-94.ec2.internal\" DevicePath \"\""
Jul 12 17:05:33.982716 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:33.982530    2569 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea-secret-prometheus-k8s-tls\") on node \"ip-10-0-141-94.ec2.internal\" DevicePath \"\""
Jul 12 17:05:33.982716 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:33.982540    2569 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea-secret-grpc-tls\") on node \"ip-10-0-141-94.ec2.internal\" DevicePath \"\""
Jul 12 17:05:33.982716 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:33.982547    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxfwl\" (UniqueName: \"kubernetes.io/projected/80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea-kube-api-access-mxfwl\") on node \"ip-10-0-141-94.ec2.internal\" DevicePath \"\""
Jul 12 17:05:33.982716 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:33.982556    2569 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-141-94.ec2.internal\" DevicePath \"\""
Jul 12 17:05:33.982716 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:33.982565    2569 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea-web-config\") on node \"ip-10-0-141-94.ec2.internal\" DevicePath \"\""
Jul 12 17:05:33.982716 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:33.982574    2569 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea-prometheus-k8s-db\") on node \"ip-10-0-141-94.ec2.internal\" DevicePath \"\""
Jul 12 17:05:33.982716 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:33.982582    2569 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea-tls-assets\") on node \"ip-10-0-141-94.ec2.internal\" DevicePath \"\""
Jul 12 17:05:33.982716 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:33.982590    2569 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea-config-out\") on node \"ip-10-0-141-94.ec2.internal\" DevicePath \"\""
Jul 12 17:05:33.982716 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:33.982597    2569 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-141-94.ec2.internal\" DevicePath \"\""
Jul 12 17:05:33.982716 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:33.982605    2569 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-141-94.ec2.internal\" DevicePath \"\""
Jul 12 17:05:33.982716 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:33.982613    2569 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea-thanos-prometheus-http-client-file\") on node \"ip-10-0-141-94.ec2.internal\" DevicePath \"\""
Jul 12 17:05:33.982716 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:33.982621    2569 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea-secret-metrics-client-certs\") on node \"ip-10-0-141-94.ec2.internal\" DevicePath \"\""
Jul 12 17:05:33.982716 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:33.982630    2569 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea-prometheus-trusted-ca-bundle\") on node \"ip-10-0-141-94.ec2.internal\" DevicePath \"\""
Jul 12 17:05:34.127171 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.127100    2569 generic.go:356] "Generic (PLEG): container finished" podID="80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea" containerID="e197714d1dac18f9ca2021d8560c716caad667fea6320cc4580bcfb46655b7d0" exitCode=0
Jul 12 17:05:34.127171 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.127124    2569 generic.go:356] "Generic (PLEG): container finished" podID="80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea" containerID="cab5febf942dd06a1f64f68038217812d5c907d2e346aef30bd2b6608a80c8b6" exitCode=0
Jul 12 17:05:34.127171 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.127131    2569 generic.go:356] "Generic (PLEG): container finished" podID="80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea" containerID="f6c03ddd3e3c7cd78a99ee6800bdddac49d6e96fafa45191d36cd152b958012d" exitCode=0
Jul 12 17:05:34.127171 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.127137    2569 generic.go:356] "Generic (PLEG): container finished" podID="80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea" containerID="be6db67c15c90939cbcd0808881e2846c2bf6da6ff54086809513d1f67690f91" exitCode=0
Jul 12 17:05:34.127171 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.127143    2569 generic.go:356] "Generic (PLEG): container finished" podID="80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea" containerID="6b068d2fa8eefc331d6406452cc63366416fec682eb2dba4c04fac9cb9789edf" exitCode=0
Jul 12 17:05:34.127171 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.127148    2569 generic.go:356] "Generic (PLEG): container finished" podID="80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea" containerID="ee358c37ed01d72a985ffbc691b2ff16825a541b53661cec266cbeb8e6d66790" exitCode=0
Jul 12 17:05:34.127420 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.127178    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 12 17:05:34.127420 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.127187    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea","Type":"ContainerDied","Data":"e197714d1dac18f9ca2021d8560c716caad667fea6320cc4580bcfb46655b7d0"}
Jul 12 17:05:34.127420 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.127261    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea","Type":"ContainerDied","Data":"cab5febf942dd06a1f64f68038217812d5c907d2e346aef30bd2b6608a80c8b6"}
Jul 12 17:05:34.127420 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.127291    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea","Type":"ContainerDied","Data":"f6c03ddd3e3c7cd78a99ee6800bdddac49d6e96fafa45191d36cd152b958012d"}
Jul 12 17:05:34.127420 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.127326    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea","Type":"ContainerDied","Data":"be6db67c15c90939cbcd0808881e2846c2bf6da6ff54086809513d1f67690f91"}
Jul 12 17:05:34.127420 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.127331    2569 scope.go:117] "RemoveContainer" containerID="e197714d1dac18f9ca2021d8560c716caad667fea6320cc4580bcfb46655b7d0"
Jul 12 17:05:34.127420 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.127365    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea","Type":"ContainerDied","Data":"6b068d2fa8eefc331d6406452cc63366416fec682eb2dba4c04fac9cb9789edf"}
Jul 12 17:05:34.127420 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.127405    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea","Type":"ContainerDied","Data":"ee358c37ed01d72a985ffbc691b2ff16825a541b53661cec266cbeb8e6d66790"}
Jul 12 17:05:34.127744 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.127443    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea","Type":"ContainerDied","Data":"f4fe3dbde4dcc094953caa40ef5b018c1b6d3e945c805b24863b25bc48c62eeb"}
Jul 12 17:05:34.134933 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.134903    2569 scope.go:117] "RemoveContainer" containerID="cab5febf942dd06a1f64f68038217812d5c907d2e346aef30bd2b6608a80c8b6"
Jul 12 17:05:34.141607 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.141579    2569 scope.go:117] "RemoveContainer" containerID="f6c03ddd3e3c7cd78a99ee6800bdddac49d6e96fafa45191d36cd152b958012d"
Jul 12 17:05:34.147673 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.147658    2569 scope.go:117] "RemoveContainer" containerID="be6db67c15c90939cbcd0808881e2846c2bf6da6ff54086809513d1f67690f91"
Jul 12 17:05:34.153321 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.153298    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 12 17:05:34.153868 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.153845    2569 scope.go:117] "RemoveContainer" containerID="6b068d2fa8eefc331d6406452cc63366416fec682eb2dba4c04fac9cb9789edf"
Jul 12 17:05:34.160279 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.160259    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 12 17:05:34.160611 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.160591    2569 scope.go:117] "RemoveContainer" containerID="ee358c37ed01d72a985ffbc691b2ff16825a541b53661cec266cbeb8e6d66790"
Jul 12 17:05:34.167060 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.167040    2569 scope.go:117] "RemoveContainer" containerID="c7a9d708b26bace74a6060fc9a225c798971d4747c63cd4c540571c92222036d"
Jul 12 17:05:34.173174 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.173159    2569 scope.go:117] "RemoveContainer" containerID="e197714d1dac18f9ca2021d8560c716caad667fea6320cc4580bcfb46655b7d0"
Jul 12 17:05:34.173413 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:05:34.173397    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e197714d1dac18f9ca2021d8560c716caad667fea6320cc4580bcfb46655b7d0\": container with ID starting with e197714d1dac18f9ca2021d8560c716caad667fea6320cc4580bcfb46655b7d0 not found: ID does not exist" containerID="e197714d1dac18f9ca2021d8560c716caad667fea6320cc4580bcfb46655b7d0"
Jul 12 17:05:34.173467 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.173418    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e197714d1dac18f9ca2021d8560c716caad667fea6320cc4580bcfb46655b7d0"} err="failed to get container status \"e197714d1dac18f9ca2021d8560c716caad667fea6320cc4580bcfb46655b7d0\": rpc error: code = NotFound desc = could not find container \"e197714d1dac18f9ca2021d8560c716caad667fea6320cc4580bcfb46655b7d0\": container with ID starting with e197714d1dac18f9ca2021d8560c716caad667fea6320cc4580bcfb46655b7d0 not found: ID does not exist"
Jul 12 17:05:34.173467 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.173434    2569 scope.go:117] "RemoveContainer" containerID="cab5febf942dd06a1f64f68038217812d5c907d2e346aef30bd2b6608a80c8b6"
Jul 12 17:05:34.173669 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:05:34.173650    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cab5febf942dd06a1f64f68038217812d5c907d2e346aef30bd2b6608a80c8b6\": container with ID starting with cab5febf942dd06a1f64f68038217812d5c907d2e346aef30bd2b6608a80c8b6 not found: ID does not exist" containerID="cab5febf942dd06a1f64f68038217812d5c907d2e346aef30bd2b6608a80c8b6"
Jul 12 17:05:34.173730 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.173680    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cab5febf942dd06a1f64f68038217812d5c907d2e346aef30bd2b6608a80c8b6"} err="failed to get container status \"cab5febf942dd06a1f64f68038217812d5c907d2e346aef30bd2b6608a80c8b6\": rpc error: code = NotFound desc = could not find container \"cab5febf942dd06a1f64f68038217812d5c907d2e346aef30bd2b6608a80c8b6\": container with ID starting with cab5febf942dd06a1f64f68038217812d5c907d2e346aef30bd2b6608a80c8b6 not found: ID does not exist"
Jul 12 17:05:34.173730 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.173705    2569 scope.go:117] "RemoveContainer" containerID="f6c03ddd3e3c7cd78a99ee6800bdddac49d6e96fafa45191d36cd152b958012d"
Jul 12 17:05:34.173953 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:05:34.173936    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6c03ddd3e3c7cd78a99ee6800bdddac49d6e96fafa45191d36cd152b958012d\": container with ID starting with f6c03ddd3e3c7cd78a99ee6800bdddac49d6e96fafa45191d36cd152b958012d not found: ID does not exist" containerID="f6c03ddd3e3c7cd78a99ee6800bdddac49d6e96fafa45191d36cd152b958012d"
Jul 12 17:05:34.173988 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.173958    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6c03ddd3e3c7cd78a99ee6800bdddac49d6e96fafa45191d36cd152b958012d"} err="failed to get container status \"f6c03ddd3e3c7cd78a99ee6800bdddac49d6e96fafa45191d36cd152b958012d\": rpc error: code = NotFound desc = could not find container \"f6c03ddd3e3c7cd78a99ee6800bdddac49d6e96fafa45191d36cd152b958012d\": container with ID starting with f6c03ddd3e3c7cd78a99ee6800bdddac49d6e96fafa45191d36cd152b958012d not found: ID does not exist"
Jul 12 17:05:34.173988 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.173972    2569 scope.go:117] "RemoveContainer" containerID="be6db67c15c90939cbcd0808881e2846c2bf6da6ff54086809513d1f67690f91"
Jul 12 17:05:34.174230 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:05:34.174200    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be6db67c15c90939cbcd0808881e2846c2bf6da6ff54086809513d1f67690f91\": container with ID starting with be6db67c15c90939cbcd0808881e2846c2bf6da6ff54086809513d1f67690f91 not found: ID does not exist" containerID="be6db67c15c90939cbcd0808881e2846c2bf6da6ff54086809513d1f67690f91"
Jul 12 17:05:34.174274 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.174234    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be6db67c15c90939cbcd0808881e2846c2bf6da6ff54086809513d1f67690f91"} err="failed to get container status \"be6db67c15c90939cbcd0808881e2846c2bf6da6ff54086809513d1f67690f91\": rpc error: code = NotFound desc = could not find container \"be6db67c15c90939cbcd0808881e2846c2bf6da6ff54086809513d1f67690f91\": container with ID starting with be6db67c15c90939cbcd0808881e2846c2bf6da6ff54086809513d1f67690f91 not found: ID does not exist"
Jul 12 17:05:34.174274 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.174250    2569 scope.go:117] "RemoveContainer" containerID="6b068d2fa8eefc331d6406452cc63366416fec682eb2dba4c04fac9cb9789edf"
Jul 12 17:05:34.174464 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:05:34.174447    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b068d2fa8eefc331d6406452cc63366416fec682eb2dba4c04fac9cb9789edf\": container with ID starting with 6b068d2fa8eefc331d6406452cc63366416fec682eb2dba4c04fac9cb9789edf not found: ID does not exist" containerID="6b068d2fa8eefc331d6406452cc63366416fec682eb2dba4c04fac9cb9789edf"
Jul 12 17:05:34.174527 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.174471    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b068d2fa8eefc331d6406452cc63366416fec682eb2dba4c04fac9cb9789edf"} err="failed to get container status \"6b068d2fa8eefc331d6406452cc63366416fec682eb2dba4c04fac9cb9789edf\": rpc error: code = NotFound desc = could not find container \"6b068d2fa8eefc331d6406452cc63366416fec682eb2dba4c04fac9cb9789edf\": container with ID starting with 6b068d2fa8eefc331d6406452cc63366416fec682eb2dba4c04fac9cb9789edf not found: ID does not exist"
Jul 12 17:05:34.174527 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.174487    2569 scope.go:117] "RemoveContainer" containerID="ee358c37ed01d72a985ffbc691b2ff16825a541b53661cec266cbeb8e6d66790"
Jul 12 17:05:34.174773 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:05:34.174755    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee358c37ed01d72a985ffbc691b2ff16825a541b53661cec266cbeb8e6d66790\": container with ID starting with ee358c37ed01d72a985ffbc691b2ff16825a541b53661cec266cbeb8e6d66790 not found: ID does not exist" containerID="ee358c37ed01d72a985ffbc691b2ff16825a541b53661cec266cbeb8e6d66790"
Jul 12 17:05:34.174845 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.174781    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee358c37ed01d72a985ffbc691b2ff16825a541b53661cec266cbeb8e6d66790"} err="failed to get container status \"ee358c37ed01d72a985ffbc691b2ff16825a541b53661cec266cbeb8e6d66790\": rpc error: code = NotFound desc = could not find container \"ee358c37ed01d72a985ffbc691b2ff16825a541b53661cec266cbeb8e6d66790\": container with ID starting with ee358c37ed01d72a985ffbc691b2ff16825a541b53661cec266cbeb8e6d66790 not found: ID does not exist"
Jul 12 17:05:34.174845 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.174803    2569 scope.go:117] "RemoveContainer" containerID="c7a9d708b26bace74a6060fc9a225c798971d4747c63cd4c540571c92222036d"
Jul 12 17:05:34.175062 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:05:34.175047    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7a9d708b26bace74a6060fc9a225c798971d4747c63cd4c540571c92222036d\": container with ID starting with c7a9d708b26bace74a6060fc9a225c798971d4747c63cd4c540571c92222036d not found: ID does not exist" containerID="c7a9d708b26bace74a6060fc9a225c798971d4747c63cd4c540571c92222036d"
Jul 12 17:05:34.175100 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.175066    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7a9d708b26bace74a6060fc9a225c798971d4747c63cd4c540571c92222036d"} err="failed to get container status \"c7a9d708b26bace74a6060fc9a225c798971d4747c63cd4c540571c92222036d\": rpc error: code = NotFound desc = could not find container \"c7a9d708b26bace74a6060fc9a225c798971d4747c63cd4c540571c92222036d\": container with ID starting with c7a9d708b26bace74a6060fc9a225c798971d4747c63cd4c540571c92222036d not found: ID does not exist"
Jul 12 17:05:34.175100 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.175080    2569 scope.go:117] "RemoveContainer" containerID="e197714d1dac18f9ca2021d8560c716caad667fea6320cc4580bcfb46655b7d0"
Jul 12 17:05:34.175300 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.175283    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e197714d1dac18f9ca2021d8560c716caad667fea6320cc4580bcfb46655b7d0"} err="failed to get container status \"e197714d1dac18f9ca2021d8560c716caad667fea6320cc4580bcfb46655b7d0\": rpc error: code = NotFound desc = could not find container \"e197714d1dac18f9ca2021d8560c716caad667fea6320cc4580bcfb46655b7d0\": container with ID starting with e197714d1dac18f9ca2021d8560c716caad667fea6320cc4580bcfb46655b7d0 not found: ID does not exist"
Jul 12 17:05:34.175348 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.175301    2569 scope.go:117] "RemoveContainer" containerID="cab5febf942dd06a1f64f68038217812d5c907d2e346aef30bd2b6608a80c8b6"
Jul 12 17:05:34.175544 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.175489    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cab5febf942dd06a1f64f68038217812d5c907d2e346aef30bd2b6608a80c8b6"} err="failed to get container status \"cab5febf942dd06a1f64f68038217812d5c907d2e346aef30bd2b6608a80c8b6\": rpc error: code = NotFound desc = could not find container \"cab5febf942dd06a1f64f68038217812d5c907d2e346aef30bd2b6608a80c8b6\": container with ID starting with cab5febf942dd06a1f64f68038217812d5c907d2e346aef30bd2b6608a80c8b6 not found: ID does not exist"
Jul 12 17:05:34.175544 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.175525    2569 scope.go:117] "RemoveContainer" containerID="f6c03ddd3e3c7cd78a99ee6800bdddac49d6e96fafa45191d36cd152b958012d"
Jul 12 17:05:34.175771 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.175755    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6c03ddd3e3c7cd78a99ee6800bdddac49d6e96fafa45191d36cd152b958012d"} err="failed to get container status \"f6c03ddd3e3c7cd78a99ee6800bdddac49d6e96fafa45191d36cd152b958012d\": rpc error: code = NotFound desc = could not find container \"f6c03ddd3e3c7cd78a99ee6800bdddac49d6e96fafa45191d36cd152b958012d\": container with ID starting with f6c03ddd3e3c7cd78a99ee6800bdddac49d6e96fafa45191d36cd152b958012d not found: ID does not exist"
Jul 12 17:05:34.175817 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.175773    2569 scope.go:117] "RemoveContainer" containerID="be6db67c15c90939cbcd0808881e2846c2bf6da6ff54086809513d1f67690f91"
Jul 12 17:05:34.175973 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.175956    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be6db67c15c90939cbcd0808881e2846c2bf6da6ff54086809513d1f67690f91"} err="failed to get container status \"be6db67c15c90939cbcd0808881e2846c2bf6da6ff54086809513d1f67690f91\": rpc error: code = NotFound desc = could not find container \"be6db67c15c90939cbcd0808881e2846c2bf6da6ff54086809513d1f67690f91\": container with ID starting with be6db67c15c90939cbcd0808881e2846c2bf6da6ff54086809513d1f67690f91 not found: ID does not exist"
Jul 12 17:05:34.176031 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.175973    2569 scope.go:117] "RemoveContainer" containerID="6b068d2fa8eefc331d6406452cc63366416fec682eb2dba4c04fac9cb9789edf"
Jul 12 17:05:34.176194 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.176174    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b068d2fa8eefc331d6406452cc63366416fec682eb2dba4c04fac9cb9789edf"} err="failed to get container status \"6b068d2fa8eefc331d6406452cc63366416fec682eb2dba4c04fac9cb9789edf\": rpc error: code = NotFound desc = could not find container \"6b068d2fa8eefc331d6406452cc63366416fec682eb2dba4c04fac9cb9789edf\": container with ID starting with 6b068d2fa8eefc331d6406452cc63366416fec682eb2dba4c04fac9cb9789edf not found: ID does not exist"
Jul 12 17:05:34.176256 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.176197    2569 scope.go:117] "RemoveContainer" containerID="ee358c37ed01d72a985ffbc691b2ff16825a541b53661cec266cbeb8e6d66790"
Jul 12 17:05:34.176390 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.176371    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee358c37ed01d72a985ffbc691b2ff16825a541b53661cec266cbeb8e6d66790"} err="failed to get container status \"ee358c37ed01d72a985ffbc691b2ff16825a541b53661cec266cbeb8e6d66790\": rpc error: code = NotFound desc = could not find container \"ee358c37ed01d72a985ffbc691b2ff16825a541b53661cec266cbeb8e6d66790\": container with ID starting with ee358c37ed01d72a985ffbc691b2ff16825a541b53661cec266cbeb8e6d66790 not found: ID does not exist"
Jul 12 17:05:34.176452 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.176390    2569 scope.go:117] "RemoveContainer" containerID="c7a9d708b26bace74a6060fc9a225c798971d4747c63cd4c540571c92222036d"
Jul 12 17:05:34.176584 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.176566    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7a9d708b26bace74a6060fc9a225c798971d4747c63cd4c540571c92222036d"} err="failed to get container status \"c7a9d708b26bace74a6060fc9a225c798971d4747c63cd4c540571c92222036d\": rpc error: code = NotFound desc = could not find container \"c7a9d708b26bace74a6060fc9a225c798971d4747c63cd4c540571c92222036d\": container with ID starting with c7a9d708b26bace74a6060fc9a225c798971d4747c63cd4c540571c92222036d not found: ID does not exist"
Jul 12 17:05:34.176633 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.176584    2569 scope.go:117] "RemoveContainer" containerID="e197714d1dac18f9ca2021d8560c716caad667fea6320cc4580bcfb46655b7d0"
Jul 12 17:05:34.176788 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.176769    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e197714d1dac18f9ca2021d8560c716caad667fea6320cc4580bcfb46655b7d0"} err="failed to get container status \"e197714d1dac18f9ca2021d8560c716caad667fea6320cc4580bcfb46655b7d0\": rpc error: code = NotFound desc = could not find container \"e197714d1dac18f9ca2021d8560c716caad667fea6320cc4580bcfb46655b7d0\": container with ID starting with e197714d1dac18f9ca2021d8560c716caad667fea6320cc4580bcfb46655b7d0 not found: ID does not exist"
Jul 12 17:05:34.176854 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.176790    2569 scope.go:117] "RemoveContainer" containerID="cab5febf942dd06a1f64f68038217812d5c907d2e346aef30bd2b6608a80c8b6"
Jul 12 17:05:34.177010 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.176993    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cab5febf942dd06a1f64f68038217812d5c907d2e346aef30bd2b6608a80c8b6"} err="failed to get container status \"cab5febf942dd06a1f64f68038217812d5c907d2e346aef30bd2b6608a80c8b6\": rpc error: code = NotFound desc = could not find container \"cab5febf942dd06a1f64f68038217812d5c907d2e346aef30bd2b6608a80c8b6\": container with ID starting with cab5febf942dd06a1f64f68038217812d5c907d2e346aef30bd2b6608a80c8b6 not found: ID does not exist"
Jul 12 17:05:34.177075 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.177012    2569 scope.go:117] "RemoveContainer" containerID="f6c03ddd3e3c7cd78a99ee6800bdddac49d6e96fafa45191d36cd152b958012d"
Jul 12 17:05:34.177214 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.177195    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6c03ddd3e3c7cd78a99ee6800bdddac49d6e96fafa45191d36cd152b958012d"} err="failed to get container status \"f6c03ddd3e3c7cd78a99ee6800bdddac49d6e96fafa45191d36cd152b958012d\": rpc error: code = NotFound desc = could not find container \"f6c03ddd3e3c7cd78a99ee6800bdddac49d6e96fafa45191d36cd152b958012d\": container with ID starting with f6c03ddd3e3c7cd78a99ee6800bdddac49d6e96fafa45191d36cd152b958012d not found: ID does not exist"
Jul 12 17:05:34.177281 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.177218    2569 scope.go:117] "RemoveContainer" containerID="be6db67c15c90939cbcd0808881e2846c2bf6da6ff54086809513d1f67690f91"
Jul 12 17:05:34.177412 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.177395    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be6db67c15c90939cbcd0808881e2846c2bf6da6ff54086809513d1f67690f91"} err="failed to get container status \"be6db67c15c90939cbcd0808881e2846c2bf6da6ff54086809513d1f67690f91\": rpc error: code = NotFound desc = could not find container \"be6db67c15c90939cbcd0808881e2846c2bf6da6ff54086809513d1f67690f91\": container with ID starting with be6db67c15c90939cbcd0808881e2846c2bf6da6ff54086809513d1f67690f91 not found: ID does not exist"
Jul 12 17:05:34.177476 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.177414    2569 scope.go:117] "RemoveContainer" containerID="6b068d2fa8eefc331d6406452cc63366416fec682eb2dba4c04fac9cb9789edf"
Jul 12 17:05:34.177632 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.177612    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b068d2fa8eefc331d6406452cc63366416fec682eb2dba4c04fac9cb9789edf"} err="failed to get container status \"6b068d2fa8eefc331d6406452cc63366416fec682eb2dba4c04fac9cb9789edf\": rpc error: code = NotFound desc = could not find container \"6b068d2fa8eefc331d6406452cc63366416fec682eb2dba4c04fac9cb9789edf\": container with ID starting with 6b068d2fa8eefc331d6406452cc63366416fec682eb2dba4c04fac9cb9789edf not found: ID does not exist"
Jul 12 17:05:34.177679 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.177633    2569 scope.go:117] "RemoveContainer" containerID="ee358c37ed01d72a985ffbc691b2ff16825a541b53661cec266cbeb8e6d66790"
Jul 12 17:05:34.177823 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.177807    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee358c37ed01d72a985ffbc691b2ff16825a541b53661cec266cbeb8e6d66790"} err="failed to get container status \"ee358c37ed01d72a985ffbc691b2ff16825a541b53661cec266cbeb8e6d66790\": rpc error: code = NotFound desc = could not find container \"ee358c37ed01d72a985ffbc691b2ff16825a541b53661cec266cbeb8e6d66790\": container with ID starting with ee358c37ed01d72a985ffbc691b2ff16825a541b53661cec266cbeb8e6d66790 not found: ID does not exist"
Jul 12 17:05:34.177860 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.177823    2569 scope.go:117] "RemoveContainer" containerID="c7a9d708b26bace74a6060fc9a225c798971d4747c63cd4c540571c92222036d"
Jul 12 17:05:34.178080 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.178058    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7a9d708b26bace74a6060fc9a225c798971d4747c63cd4c540571c92222036d"} err="failed to get container status \"c7a9d708b26bace74a6060fc9a225c798971d4747c63cd4c540571c92222036d\": rpc error: code = NotFound desc = could not find container \"c7a9d708b26bace74a6060fc9a225c798971d4747c63cd4c540571c92222036d\": container with ID starting with c7a9d708b26bace74a6060fc9a225c798971d4747c63cd4c540571c92222036d not found: ID does not exist"
Jul 12 17:05:34.178159 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.178082    2569 scope.go:117] "RemoveContainer" containerID="e197714d1dac18f9ca2021d8560c716caad667fea6320cc4580bcfb46655b7d0"
Jul 12 17:05:34.178311 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.178287    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e197714d1dac18f9ca2021d8560c716caad667fea6320cc4580bcfb46655b7d0"} err="failed to get container status \"e197714d1dac18f9ca2021d8560c716caad667fea6320cc4580bcfb46655b7d0\": rpc error: code = NotFound desc = could not find container \"e197714d1dac18f9ca2021d8560c716caad667fea6320cc4580bcfb46655b7d0\": container with ID starting with e197714d1dac18f9ca2021d8560c716caad667fea6320cc4580bcfb46655b7d0 not found: ID does not exist"
Jul 12 17:05:34.178311 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.178310    2569 scope.go:117] "RemoveContainer" containerID="cab5febf942dd06a1f64f68038217812d5c907d2e346aef30bd2b6608a80c8b6"
Jul 12 17:05:34.178574 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.178553    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cab5febf942dd06a1f64f68038217812d5c907d2e346aef30bd2b6608a80c8b6"} err="failed to get container status \"cab5febf942dd06a1f64f68038217812d5c907d2e346aef30bd2b6608a80c8b6\": rpc error: code = NotFound desc = could not find container \"cab5febf942dd06a1f64f68038217812d5c907d2e346aef30bd2b6608a80c8b6\": container with ID starting with cab5febf942dd06a1f64f68038217812d5c907d2e346aef30bd2b6608a80c8b6 not found: ID does not exist"
Jul 12 17:05:34.178651 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.178577    2569 scope.go:117] "RemoveContainer" containerID="f6c03ddd3e3c7cd78a99ee6800bdddac49d6e96fafa45191d36cd152b958012d"
Jul 12 17:05:34.178795 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.178777    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6c03ddd3e3c7cd78a99ee6800bdddac49d6e96fafa45191d36cd152b958012d"} err="failed to get container status \"f6c03ddd3e3c7cd78a99ee6800bdddac49d6e96fafa45191d36cd152b958012d\": rpc error: code = NotFound desc = could not find container \"f6c03ddd3e3c7cd78a99ee6800bdddac49d6e96fafa45191d36cd152b958012d\": container with ID starting with f6c03ddd3e3c7cd78a99ee6800bdddac49d6e96fafa45191d36cd152b958012d not found: ID does not exist"
Jul 12 17:05:34.178845 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.178796    2569 scope.go:117] "RemoveContainer" containerID="be6db67c15c90939cbcd0808881e2846c2bf6da6ff54086809513d1f67690f91"
Jul 12 17:05:34.179012 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.178995    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be6db67c15c90939cbcd0808881e2846c2bf6da6ff54086809513d1f67690f91"} err="failed to get container status \"be6db67c15c90939cbcd0808881e2846c2bf6da6ff54086809513d1f67690f91\": rpc error: code = NotFound desc = could not find container \"be6db67c15c90939cbcd0808881e2846c2bf6da6ff54086809513d1f67690f91\": container with ID starting with be6db67c15c90939cbcd0808881e2846c2bf6da6ff54086809513d1f67690f91 not found: ID does not exist"
Jul 12 17:05:34.179057 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.179012    2569 scope.go:117] "RemoveContainer" containerID="6b068d2fa8eefc331d6406452cc63366416fec682eb2dba4c04fac9cb9789edf"
Jul 12 17:05:34.179227 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.179209    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b068d2fa8eefc331d6406452cc63366416fec682eb2dba4c04fac9cb9789edf"} err="failed to get container status \"6b068d2fa8eefc331d6406452cc63366416fec682eb2dba4c04fac9cb9789edf\": rpc error: code = NotFound desc = could not find container \"6b068d2fa8eefc331d6406452cc63366416fec682eb2dba4c04fac9cb9789edf\": container with ID starting with 6b068d2fa8eefc331d6406452cc63366416fec682eb2dba4c04fac9cb9789edf not found: ID does not exist"
Jul 12 17:05:34.179287 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.179228    2569 scope.go:117] "RemoveContainer" containerID="ee358c37ed01d72a985ffbc691b2ff16825a541b53661cec266cbeb8e6d66790"
Jul 12 17:05:34.179445 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.179428    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee358c37ed01d72a985ffbc691b2ff16825a541b53661cec266cbeb8e6d66790"} err="failed to get container status \"ee358c37ed01d72a985ffbc691b2ff16825a541b53661cec266cbeb8e6d66790\": rpc error: code = NotFound desc = could not find container \"ee358c37ed01d72a985ffbc691b2ff16825a541b53661cec266cbeb8e6d66790\": container with ID starting with ee358c37ed01d72a985ffbc691b2ff16825a541b53661cec266cbeb8e6d66790 not found: ID does not exist"
Jul 12 17:05:34.179490 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.179446    2569 scope.go:117] "RemoveContainer" containerID="c7a9d708b26bace74a6060fc9a225c798971d4747c63cd4c540571c92222036d"
Jul 12 17:05:34.179667 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.179651    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7a9d708b26bace74a6060fc9a225c798971d4747c63cd4c540571c92222036d"} err="failed to get container status \"c7a9d708b26bace74a6060fc9a225c798971d4747c63cd4c540571c92222036d\": rpc error: code = NotFound desc = could not find container \"c7a9d708b26bace74a6060fc9a225c798971d4747c63cd4c540571c92222036d\": container with ID starting with c7a9d708b26bace74a6060fc9a225c798971d4747c63cd4c540571c92222036d not found: ID does not exist"
Jul 12 17:05:34.179715 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.179668    2569 scope.go:117] "RemoveContainer" containerID="e197714d1dac18f9ca2021d8560c716caad667fea6320cc4580bcfb46655b7d0"
Jul 12 17:05:34.179885 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.179867    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e197714d1dac18f9ca2021d8560c716caad667fea6320cc4580bcfb46655b7d0"} err="failed to get container status \"e197714d1dac18f9ca2021d8560c716caad667fea6320cc4580bcfb46655b7d0\": rpc error: code = NotFound desc = could not find container \"e197714d1dac18f9ca2021d8560c716caad667fea6320cc4580bcfb46655b7d0\": container with ID starting with e197714d1dac18f9ca2021d8560c716caad667fea6320cc4580bcfb46655b7d0 not found: ID does not exist"
Jul 12 17:05:34.179932 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.179886    2569 scope.go:117] "RemoveContainer" containerID="cab5febf942dd06a1f64f68038217812d5c907d2e346aef30bd2b6608a80c8b6"
Jul 12 17:05:34.180116 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.180097    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cab5febf942dd06a1f64f68038217812d5c907d2e346aef30bd2b6608a80c8b6"} err="failed to get container status \"cab5febf942dd06a1f64f68038217812d5c907d2e346aef30bd2b6608a80c8b6\": rpc error: code = NotFound desc = could not find container \"cab5febf942dd06a1f64f68038217812d5c907d2e346aef30bd2b6608a80c8b6\": container with ID starting with cab5febf942dd06a1f64f68038217812d5c907d2e346aef30bd2b6608a80c8b6 not found: ID does not exist"
Jul 12 17:05:34.180187 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.180118    2569 scope.go:117] "RemoveContainer" containerID="f6c03ddd3e3c7cd78a99ee6800bdddac49d6e96fafa45191d36cd152b958012d"
Jul 12 17:05:34.180361 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.180344    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6c03ddd3e3c7cd78a99ee6800bdddac49d6e96fafa45191d36cd152b958012d"} err="failed to get container status \"f6c03ddd3e3c7cd78a99ee6800bdddac49d6e96fafa45191d36cd152b958012d\": rpc error: code = NotFound desc = could not find container \"f6c03ddd3e3c7cd78a99ee6800bdddac49d6e96fafa45191d36cd152b958012d\": container with ID starting with f6c03ddd3e3c7cd78a99ee6800bdddac49d6e96fafa45191d36cd152b958012d not found: ID does not exist"
Jul 12 17:05:34.180404 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.180362    2569 scope.go:117] "RemoveContainer" containerID="be6db67c15c90939cbcd0808881e2846c2bf6da6ff54086809513d1f67690f91"
Jul 12 17:05:34.180576 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.180559    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be6db67c15c90939cbcd0808881e2846c2bf6da6ff54086809513d1f67690f91"} err="failed to get container status \"be6db67c15c90939cbcd0808881e2846c2bf6da6ff54086809513d1f67690f91\": rpc error: code = NotFound desc = could not find container \"be6db67c15c90939cbcd0808881e2846c2bf6da6ff54086809513d1f67690f91\": container with ID starting with be6db67c15c90939cbcd0808881e2846c2bf6da6ff54086809513d1f67690f91 not found: ID does not exist"
Jul 12 17:05:34.180629 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.180577    2569 scope.go:117] "RemoveContainer" containerID="6b068d2fa8eefc331d6406452cc63366416fec682eb2dba4c04fac9cb9789edf"
Jul 12 17:05:34.180773 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.180755    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b068d2fa8eefc331d6406452cc63366416fec682eb2dba4c04fac9cb9789edf"} err="failed to get container status \"6b068d2fa8eefc331d6406452cc63366416fec682eb2dba4c04fac9cb9789edf\": rpc error: code = NotFound desc = could not find container \"6b068d2fa8eefc331d6406452cc63366416fec682eb2dba4c04fac9cb9789edf\": container with ID starting with 6b068d2fa8eefc331d6406452cc63366416fec682eb2dba4c04fac9cb9789edf not found: ID does not exist"
Jul 12 17:05:34.180816 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.180774    2569 scope.go:117] "RemoveContainer" containerID="ee358c37ed01d72a985ffbc691b2ff16825a541b53661cec266cbeb8e6d66790"
Jul 12 17:05:34.180967 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.180950    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee358c37ed01d72a985ffbc691b2ff16825a541b53661cec266cbeb8e6d66790"} err="failed to get container status \"ee358c37ed01d72a985ffbc691b2ff16825a541b53661cec266cbeb8e6d66790\": rpc error: code = NotFound desc = could not find container \"ee358c37ed01d72a985ffbc691b2ff16825a541b53661cec266cbeb8e6d66790\": container with ID starting with ee358c37ed01d72a985ffbc691b2ff16825a541b53661cec266cbeb8e6d66790 not found: ID does not exist"
Jul 12 17:05:34.181033 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.180969    2569 scope.go:117] "RemoveContainer" containerID="c7a9d708b26bace74a6060fc9a225c798971d4747c63cd4c540571c92222036d"
Jul 12 17:05:34.181177 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.181160    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7a9d708b26bace74a6060fc9a225c798971d4747c63cd4c540571c92222036d"} err="failed to get container status \"c7a9d708b26bace74a6060fc9a225c798971d4747c63cd4c540571c92222036d\": rpc error: code = NotFound desc = could not find container \"c7a9d708b26bace74a6060fc9a225c798971d4747c63cd4c540571c92222036d\": container with ID starting with c7a9d708b26bace74a6060fc9a225c798971d4747c63cd4c540571c92222036d not found: ID does not exist"
Jul 12 17:05:34.181219 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.181177    2569 scope.go:117] "RemoveContainer" containerID="e197714d1dac18f9ca2021d8560c716caad667fea6320cc4580bcfb46655b7d0"
Jul 12 17:05:34.181354 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.181338    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e197714d1dac18f9ca2021d8560c716caad667fea6320cc4580bcfb46655b7d0"} err="failed to get container status \"e197714d1dac18f9ca2021d8560c716caad667fea6320cc4580bcfb46655b7d0\": rpc error: code = NotFound desc = could not find container \"e197714d1dac18f9ca2021d8560c716caad667fea6320cc4580bcfb46655b7d0\": container with ID starting with e197714d1dac18f9ca2021d8560c716caad667fea6320cc4580bcfb46655b7d0 not found: ID does not exist"
Jul 12 17:05:34.181416 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.181355    2569 scope.go:117] "RemoveContainer" containerID="cab5febf942dd06a1f64f68038217812d5c907d2e346aef30bd2b6608a80c8b6"
Jul 12 17:05:34.181563 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.181547    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cab5febf942dd06a1f64f68038217812d5c907d2e346aef30bd2b6608a80c8b6"} err="failed to get container status \"cab5febf942dd06a1f64f68038217812d5c907d2e346aef30bd2b6608a80c8b6\": rpc error: code = NotFound desc = could not find container \"cab5febf942dd06a1f64f68038217812d5c907d2e346aef30bd2b6608a80c8b6\": container with ID starting with cab5febf942dd06a1f64f68038217812d5c907d2e346aef30bd2b6608a80c8b6 not found: ID does not exist"
Jul 12 17:05:34.181611 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.181563    2569 scope.go:117] "RemoveContainer" containerID="f6c03ddd3e3c7cd78a99ee6800bdddac49d6e96fafa45191d36cd152b958012d"
Jul 12 17:05:34.181748 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.181731    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6c03ddd3e3c7cd78a99ee6800bdddac49d6e96fafa45191d36cd152b958012d"} err="failed to get container status \"f6c03ddd3e3c7cd78a99ee6800bdddac49d6e96fafa45191d36cd152b958012d\": rpc error: code = NotFound desc = could not find container \"f6c03ddd3e3c7cd78a99ee6800bdddac49d6e96fafa45191d36cd152b958012d\": container with ID starting with f6c03ddd3e3c7cd78a99ee6800bdddac49d6e96fafa45191d36cd152b958012d not found: ID does not exist"
Jul 12 17:05:34.181816 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.181750    2569 scope.go:117] "RemoveContainer" containerID="be6db67c15c90939cbcd0808881e2846c2bf6da6ff54086809513d1f67690f91"
Jul 12 17:05:34.181935 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.181920    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be6db67c15c90939cbcd0808881e2846c2bf6da6ff54086809513d1f67690f91"} err="failed to get container status \"be6db67c15c90939cbcd0808881e2846c2bf6da6ff54086809513d1f67690f91\": rpc error: code = NotFound desc = could not find container \"be6db67c15c90939cbcd0808881e2846c2bf6da6ff54086809513d1f67690f91\": container with ID starting with be6db67c15c90939cbcd0808881e2846c2bf6da6ff54086809513d1f67690f91 not found: ID does not exist"
Jul 12 17:05:34.181979 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.181938    2569 scope.go:117] "RemoveContainer" containerID="6b068d2fa8eefc331d6406452cc63366416fec682eb2dba4c04fac9cb9789edf"
Jul 12 17:05:34.182114 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.182097    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b068d2fa8eefc331d6406452cc63366416fec682eb2dba4c04fac9cb9789edf"} err="failed to get container status \"6b068d2fa8eefc331d6406452cc63366416fec682eb2dba4c04fac9cb9789edf\": rpc error: code = NotFound desc = could not find container \"6b068d2fa8eefc331d6406452cc63366416fec682eb2dba4c04fac9cb9789edf\": container with ID starting with 6b068d2fa8eefc331d6406452cc63366416fec682eb2dba4c04fac9cb9789edf not found: ID does not exist"
Jul 12 17:05:34.182177 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.182115    2569 scope.go:117] "RemoveContainer" containerID="ee358c37ed01d72a985ffbc691b2ff16825a541b53661cec266cbeb8e6d66790"
Jul 12 17:05:34.182325 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.182311    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee358c37ed01d72a985ffbc691b2ff16825a541b53661cec266cbeb8e6d66790"} err="failed to get container status \"ee358c37ed01d72a985ffbc691b2ff16825a541b53661cec266cbeb8e6d66790\": rpc error: code = NotFound desc = could not find container \"ee358c37ed01d72a985ffbc691b2ff16825a541b53661cec266cbeb8e6d66790\": container with ID starting with ee358c37ed01d72a985ffbc691b2ff16825a541b53661cec266cbeb8e6d66790 not found: ID does not exist"
Jul 12 17:05:34.182372 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.182325    2569 scope.go:117] "RemoveContainer" containerID="c7a9d708b26bace74a6060fc9a225c798971d4747c63cd4c540571c92222036d"
Jul 12 17:05:34.182531 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.182497    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7a9d708b26bace74a6060fc9a225c798971d4747c63cd4c540571c92222036d"} err="failed to get container status \"c7a9d708b26bace74a6060fc9a225c798971d4747c63cd4c540571c92222036d\": rpc error: code = NotFound desc = could not find container \"c7a9d708b26bace74a6060fc9a225c798971d4747c63cd4c540571c92222036d\": container with ID starting with c7a9d708b26bace74a6060fc9a225c798971d4747c63cd4c540571c92222036d not found: ID does not exist"
Jul 12 17:05:34.187340 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.187322    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 12 17:05:34.187576 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.187563    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3cec2e0b-d740-4361-b911-222469a7e0a8" containerName="kube-rbac-proxy"
Jul 12 17:05:34.187624 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.187578    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="3cec2e0b-d740-4361-b911-222469a7e0a8" containerName="kube-rbac-proxy"
Jul 12 17:05:34.187624 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.187585    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea" containerName="kube-rbac-proxy-thanos"
Jul 12 17:05:34.187624 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.187591    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea" containerName="kube-rbac-proxy-thanos"
Jul 12 17:05:34.187624 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.187598    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3cec2e0b-d740-4361-b911-222469a7e0a8" containerName="alertmanager"
Jul 12 17:05:34.187624 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.187604    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="3cec2e0b-d740-4361-b911-222469a7e0a8" containerName="alertmanager"
Jul 12 17:05:34.187624 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.187612    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea" containerName="kube-rbac-proxy"
Jul 12 17:05:34.187624 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.187617    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea" containerName="kube-rbac-proxy"
Jul 12 17:05:34.187624 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.187624    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3cec2e0b-d740-4361-b911-222469a7e0a8" containerName="config-reloader"
Jul 12 17:05:34.187847 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.187629    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="3cec2e0b-d740-4361-b911-222469a7e0a8" containerName="config-reloader"
Jul 12 17:05:34.187847 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.187635    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea" containerName="init-config-reloader"
Jul 12 17:05:34.187847 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.187640    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea" containerName="init-config-reloader"
Jul 12 17:05:34.187847 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.187646    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea" containerName="thanos-sidecar"
Jul 12 17:05:34.187847 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.187651    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea" containerName="thanos-sidecar"
Jul 12 17:05:34.187847 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.187656    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3cec2e0b-d740-4361-b911-222469a7e0a8" containerName="init-config-reloader"
Jul 12 17:05:34.187847 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.187661    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="3cec2e0b-d740-4361-b911-222469a7e0a8" containerName="init-config-reloader"
Jul 12 17:05:34.187847 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.187667    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3cec2e0b-d740-4361-b911-222469a7e0a8" containerName="kube-rbac-proxy-web"
Jul 12 17:05:34.187847 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.187671    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="3cec2e0b-d740-4361-b911-222469a7e0a8" containerName="kube-rbac-proxy-web"
Jul 12 17:05:34.187847 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.187676    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea" containerName="prometheus"
Jul 12 17:05:34.187847 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.187682    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea" containerName="prometheus"
Jul 12 17:05:34.187847 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.187688    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3cec2e0b-d740-4361-b911-222469a7e0a8" containerName="kube-rbac-proxy-metric"
Jul 12 17:05:34.187847 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.187693    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="3cec2e0b-d740-4361-b911-222469a7e0a8" containerName="kube-rbac-proxy-metric"
Jul 12 17:05:34.187847 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.187700    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37783a93-d722-4cb5-b5d3-7e1cb8726990" containerName="console"
Jul 12 17:05:34.187847 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.187704    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="37783a93-d722-4cb5-b5d3-7e1cb8726990" containerName="console"
Jul 12 17:05:34.187847 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.187710    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3cec2e0b-d740-4361-b911-222469a7e0a8" containerName="prom-label-proxy"
Jul 12 17:05:34.187847 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.187715    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="3cec2e0b-d740-4361-b911-222469a7e0a8" containerName="prom-label-proxy"
Jul 12 17:05:34.187847 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.187721    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4d58845-a8df-44ff-ada6-9e794ad37f51" containerName="console"
Jul 12 17:05:34.187847 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.187725    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4d58845-a8df-44ff-ada6-9e794ad37f51" containerName="console"
Jul 12 17:05:34.187847 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.187733    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea" containerName="kube-rbac-proxy-web"
Jul 12 17:05:34.187847 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.187737    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea" containerName="kube-rbac-proxy-web"
Jul 12 17:05:34.187847 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.187743    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea" containerName="config-reloader"
Jul 12 17:05:34.187847 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.187748    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea" containerName="config-reloader"
Jul 12 17:05:34.187847 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.187785    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="3cec2e0b-d740-4361-b911-222469a7e0a8" containerName="config-reloader"
Jul 12 17:05:34.187847 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.187792    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="3cec2e0b-d740-4361-b911-222469a7e0a8" containerName="kube-rbac-proxy"
Jul 12 17:05:34.187847 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.187797    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea" containerName="prometheus"
Jul 12 17:05:34.187847 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.187802    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea" containerName="kube-rbac-proxy-thanos"
Jul 12 17:05:34.187847 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.187807    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="3cec2e0b-d740-4361-b911-222469a7e0a8" containerName="kube-rbac-proxy-web"
Jul 12 17:05:34.187847 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.187812    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="3cec2e0b-d740-4361-b911-222469a7e0a8" containerName="kube-rbac-proxy-metric"
Jul 12 17:05:34.187847 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.187817    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="3cec2e0b-d740-4361-b911-222469a7e0a8" containerName="prom-label-proxy"
Jul 12 17:05:34.187847 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.187822    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea" containerName="thanos-sidecar"
Jul 12 17:05:34.187847 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.187827    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="37783a93-d722-4cb5-b5d3-7e1cb8726990" containerName="console"
Jul 12 17:05:34.187847 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.187831    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4d58845-a8df-44ff-ada6-9e794ad37f51" containerName="console"
Jul 12 17:05:34.187847 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.187836    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="3cec2e0b-d740-4361-b911-222469a7e0a8" containerName="alertmanager"
Jul 12 17:05:34.187847 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.187840    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea" containerName="config-reloader"
Jul 12 17:05:34.187847 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.187845    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea" containerName="kube-rbac-proxy-web"
Jul 12 17:05:34.187847 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.187849    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea" containerName="kube-rbac-proxy"
Jul 12 17:05:34.191576 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.191552    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 12 17:05:34.194694 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.194442    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 12 17:05:34.194694 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.194454    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 12 17:05:34.194694 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.194493    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 12 17:05:34.194694 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.194524    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 12 17:05:34.194694 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.194546    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 12 17:05:34.194694 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.194582    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 12 17:05:34.194694 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.194629    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 12 17:05:34.194694 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.194683    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 12 17:05:34.194694 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.194703    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 12 17:05:34.195170 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.194687    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-62xzw\""
Jul 12 17:05:34.195170 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.194905    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 12 17:05:34.195280 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.195187    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-3hs5rhd03u095\""
Jul 12 17:05:34.195327 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.195303    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 12 17:05:34.195533 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.195497    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 12 17:05:34.197470 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.197450    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 12 17:05:34.199193 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.199172    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 12 17:05:34.211694 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.211675    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 12 17:05:34.284089 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.284063    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/88eea35c-c9d1-4c86-8289-9fba5f31f6fd-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"88eea35c-c9d1-4c86-8289-9fba5f31f6fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 12 17:05:34.284180 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.284093    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/88eea35c-c9d1-4c86-8289-9fba5f31f6fd-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"88eea35c-c9d1-4c86-8289-9fba5f31f6fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 12 17:05:34.284180 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.284116    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/88eea35c-c9d1-4c86-8289-9fba5f31f6fd-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"88eea35c-c9d1-4c86-8289-9fba5f31f6fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 12 17:05:34.284180 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.284137    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/88eea35c-c9d1-4c86-8289-9fba5f31f6fd-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"88eea35c-c9d1-4c86-8289-9fba5f31f6fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 12 17:05:34.284351 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.284206    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/88eea35c-c9d1-4c86-8289-9fba5f31f6fd-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"88eea35c-c9d1-4c86-8289-9fba5f31f6fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 12 17:05:34.284351 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.284247    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/88eea35c-c9d1-4c86-8289-9fba5f31f6fd-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"88eea35c-c9d1-4c86-8289-9fba5f31f6fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 12 17:05:34.284351 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.284300    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/88eea35c-c9d1-4c86-8289-9fba5f31f6fd-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"88eea35c-c9d1-4c86-8289-9fba5f31f6fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 12 17:05:34.284351 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.284330    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/88eea35c-c9d1-4c86-8289-9fba5f31f6fd-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"88eea35c-c9d1-4c86-8289-9fba5f31f6fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 12 17:05:34.284495 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.284368    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvz2k\" (UniqueName: \"kubernetes.io/projected/88eea35c-c9d1-4c86-8289-9fba5f31f6fd-kube-api-access-gvz2k\") pod \"prometheus-k8s-0\" (UID: \"88eea35c-c9d1-4c86-8289-9fba5f31f6fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 12 17:05:34.284495 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.284410    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/88eea35c-c9d1-4c86-8289-9fba5f31f6fd-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"88eea35c-c9d1-4c86-8289-9fba5f31f6fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 12 17:05:34.284495 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.284462    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/88eea35c-c9d1-4c86-8289-9fba5f31f6fd-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"88eea35c-c9d1-4c86-8289-9fba5f31f6fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 12 17:05:34.284495 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.284478    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/88eea35c-c9d1-4c86-8289-9fba5f31f6fd-web-config\") pod \"prometheus-k8s-0\" (UID: \"88eea35c-c9d1-4c86-8289-9fba5f31f6fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 12 17:05:34.284650 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.284495    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/88eea35c-c9d1-4c86-8289-9fba5f31f6fd-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"88eea35c-c9d1-4c86-8289-9fba5f31f6fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 12 17:05:34.284650 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.284531    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/88eea35c-c9d1-4c86-8289-9fba5f31f6fd-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"88eea35c-c9d1-4c86-8289-9fba5f31f6fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 12 17:05:34.284650 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.284551    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/88eea35c-c9d1-4c86-8289-9fba5f31f6fd-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"88eea35c-c9d1-4c86-8289-9fba5f31f6fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 12 17:05:34.284650 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.284584    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/88eea35c-c9d1-4c86-8289-9fba5f31f6fd-config-out\") pod \"prometheus-k8s-0\" (UID: \"88eea35c-c9d1-4c86-8289-9fba5f31f6fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 12 17:05:34.284650 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.284603    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/88eea35c-c9d1-4c86-8289-9fba5f31f6fd-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"88eea35c-c9d1-4c86-8289-9fba5f31f6fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 12 17:05:34.284650 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.284622    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/88eea35c-c9d1-4c86-8289-9fba5f31f6fd-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"88eea35c-c9d1-4c86-8289-9fba5f31f6fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 12 17:05:34.284650 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.284640    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/88eea35c-c9d1-4c86-8289-9fba5f31f6fd-config\") pod \"prometheus-k8s-0\" (UID: \"88eea35c-c9d1-4c86-8289-9fba5f31f6fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 12 17:05:34.284848 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.284666    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/88eea35c-c9d1-4c86-8289-9fba5f31f6fd-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"88eea35c-c9d1-4c86-8289-9fba5f31f6fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 12 17:05:34.378928 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.378868    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea" path="/var/lib/kubelet/pods/80fda6ca-b642-45ba-b5e7-a0b9c2bb36ea/volumes"
Jul 12 17:05:34.385678 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.385645    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/88eea35c-c9d1-4c86-8289-9fba5f31f6fd-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"88eea35c-c9d1-4c86-8289-9fba5f31f6fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 12 17:05:34.385786 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.385705    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/88eea35c-c9d1-4c86-8289-9fba5f31f6fd-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"88eea35c-c9d1-4c86-8289-9fba5f31f6fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 12 17:05:34.385786 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.385749    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvz2k\" (UniqueName: \"kubernetes.io/projected/88eea35c-c9d1-4c86-8289-9fba5f31f6fd-kube-api-access-gvz2k\") pod \"prometheus-k8s-0\" (UID: \"88eea35c-c9d1-4c86-8289-9fba5f31f6fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 12 17:05:34.385904 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.385782    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/88eea35c-c9d1-4c86-8289-9fba5f31f6fd-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"88eea35c-c9d1-4c86-8289-9fba5f31f6fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 12 17:05:34.385904 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.385809    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/88eea35c-c9d1-4c86-8289-9fba5f31f6fd-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"88eea35c-c9d1-4c86-8289-9fba5f31f6fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 12 17:05:34.385904 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.385832    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/88eea35c-c9d1-4c86-8289-9fba5f31f6fd-web-config\") pod \"prometheus-k8s-0\" (UID: \"88eea35c-c9d1-4c86-8289-9fba5f31f6fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 12 17:05:34.385904 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.385856    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/88eea35c-c9d1-4c86-8289-9fba5f31f6fd-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"88eea35c-c9d1-4c86-8289-9fba5f31f6fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 12 17:05:34.386131 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.386107    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/88eea35c-c9d1-4c86-8289-9fba5f31f6fd-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"88eea35c-c9d1-4c86-8289-9fba5f31f6fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 12 17:05:34.386189 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.386160    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/88eea35c-c9d1-4c86-8289-9fba5f31f6fd-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"88eea35c-c9d1-4c86-8289-9fba5f31f6fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 12 17:05:34.386246 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.386193    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/88eea35c-c9d1-4c86-8289-9fba5f31f6fd-config-out\") pod \"prometheus-k8s-0\" (UID: \"88eea35c-c9d1-4c86-8289-9fba5f31f6fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 12 17:05:34.386246 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.386219    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/88eea35c-c9d1-4c86-8289-9fba5f31f6fd-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"88eea35c-c9d1-4c86-8289-9fba5f31f6fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 12 17:05:34.386344 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.386252    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/88eea35c-c9d1-4c86-8289-9fba5f31f6fd-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"88eea35c-c9d1-4c86-8289-9fba5f31f6fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 12 17:05:34.386344 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.386282    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/88eea35c-c9d1-4c86-8289-9fba5f31f6fd-config\") pod \"prometheus-k8s-0\" (UID: \"88eea35c-c9d1-4c86-8289-9fba5f31f6fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 12 17:05:34.386344 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.386309    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/88eea35c-c9d1-4c86-8289-9fba5f31f6fd-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"88eea35c-c9d1-4c86-8289-9fba5f31f6fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 12 17:05:34.386344 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.386337    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/88eea35c-c9d1-4c86-8289-9fba5f31f6fd-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"88eea35c-c9d1-4c86-8289-9fba5f31f6fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 12 17:05:34.386563 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.386350    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/88eea35c-c9d1-4c86-8289-9fba5f31f6fd-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"88eea35c-c9d1-4c86-8289-9fba5f31f6fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 12 17:05:34.386563 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.386365    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/88eea35c-c9d1-4c86-8289-9fba5f31f6fd-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"88eea35c-c9d1-4c86-8289-9fba5f31f6fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 12 17:05:34.386563 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.386407    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/88eea35c-c9d1-4c86-8289-9fba5f31f6fd-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"88eea35c-c9d1-4c86-8289-9fba5f31f6fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 12 17:05:34.386563 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.386445    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/88eea35c-c9d1-4c86-8289-9fba5f31f6fd-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"88eea35c-c9d1-4c86-8289-9fba5f31f6fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 12 17:05:34.386563 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.386476    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/88eea35c-c9d1-4c86-8289-9fba5f31f6fd-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"88eea35c-c9d1-4c86-8289-9fba5f31f6fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 12 17:05:34.386563 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.386495    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/88eea35c-c9d1-4c86-8289-9fba5f31f6fd-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"88eea35c-c9d1-4c86-8289-9fba5f31f6fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 12 17:05:34.386563 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.386523    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/88eea35c-c9d1-4c86-8289-9fba5f31f6fd-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"88eea35c-c9d1-4c86-8289-9fba5f31f6fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 12 17:05:34.386901 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.386611    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/88eea35c-c9d1-4c86-8289-9fba5f31f6fd-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"88eea35c-c9d1-4c86-8289-9fba5f31f6fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 12 17:05:34.386901 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.386721    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/88eea35c-c9d1-4c86-8289-9fba5f31f6fd-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"88eea35c-c9d1-4c86-8289-9fba5f31f6fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 12 17:05:34.386901 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.386847    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/88eea35c-c9d1-4c86-8289-9fba5f31f6fd-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"88eea35c-c9d1-4c86-8289-9fba5f31f6fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 12 17:05:34.387618 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.387591    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/88eea35c-c9d1-4c86-8289-9fba5f31f6fd-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"88eea35c-c9d1-4c86-8289-9fba5f31f6fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 12 17:05:34.388075 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.388053    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/88eea35c-c9d1-4c86-8289-9fba5f31f6fd-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"88eea35c-c9d1-4c86-8289-9fba5f31f6fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 12 17:05:34.388833 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.388806    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/88eea35c-c9d1-4c86-8289-9fba5f31f6fd-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"88eea35c-c9d1-4c86-8289-9fba5f31f6fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 12 17:05:34.388915 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.388864    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/88eea35c-c9d1-4c86-8289-9fba5f31f6fd-web-config\") pod \"prometheus-k8s-0\" (UID: \"88eea35c-c9d1-4c86-8289-9fba5f31f6fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 12 17:05:34.389292 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.389191    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/88eea35c-c9d1-4c86-8289-9fba5f31f6fd-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"88eea35c-c9d1-4c86-8289-9fba5f31f6fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 12 17:05:34.389292 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.389230    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/88eea35c-c9d1-4c86-8289-9fba5f31f6fd-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"88eea35c-c9d1-4c86-8289-9fba5f31f6fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 12 17:05:34.389442 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.389360    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/88eea35c-c9d1-4c86-8289-9fba5f31f6fd-config\") pod \"prometheus-k8s-0\" (UID: \"88eea35c-c9d1-4c86-8289-9fba5f31f6fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 12 17:05:34.389722 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.389698    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/88eea35c-c9d1-4c86-8289-9fba5f31f6fd-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"88eea35c-c9d1-4c86-8289-9fba5f31f6fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 12 17:05:34.389901 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.389884    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/88eea35c-c9d1-4c86-8289-9fba5f31f6fd-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"88eea35c-c9d1-4c86-8289-9fba5f31f6fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 12 17:05:34.390413 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.390391    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/88eea35c-c9d1-4c86-8289-9fba5f31f6fd-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"88eea35c-c9d1-4c86-8289-9fba5f31f6fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 12 17:05:34.391275 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.391257    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/88eea35c-c9d1-4c86-8289-9fba5f31f6fd-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"88eea35c-c9d1-4c86-8289-9fba5f31f6fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 12 17:05:34.391405 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.391386    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/88eea35c-c9d1-4c86-8289-9fba5f31f6fd-config-out\") pod \"prometheus-k8s-0\" (UID: \"88eea35c-c9d1-4c86-8289-9fba5f31f6fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 12 17:05:34.391646 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.391627    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/88eea35c-c9d1-4c86-8289-9fba5f31f6fd-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"88eea35c-c9d1-4c86-8289-9fba5f31f6fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 12 17:05:34.392183 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.392165    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/88eea35c-c9d1-4c86-8289-9fba5f31f6fd-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"88eea35c-c9d1-4c86-8289-9fba5f31f6fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 12 17:05:34.393380 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.393364    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvz2k\" (UniqueName: \"kubernetes.io/projected/88eea35c-c9d1-4c86-8289-9fba5f31f6fd-kube-api-access-gvz2k\") pod \"prometheus-k8s-0\" (UID: \"88eea35c-c9d1-4c86-8289-9fba5f31f6fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 12 17:05:34.503559 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.503535    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 12 17:05:34.642029 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:34.642006    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 12 17:05:34.643921 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:05:34.643897    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88eea35c_c9d1_4c86_8289_9fba5f31f6fd.slice/crio-a416eef9add41dad28a04ee3360dc0b9bcf1fe433517247207ab733607714309 WatchSource:0}: Error finding container a416eef9add41dad28a04ee3360dc0b9bcf1fe433517247207ab733607714309: Status 404 returned error can't find the container with id a416eef9add41dad28a04ee3360dc0b9bcf1fe433517247207ab733607714309
Jul 12 17:05:35.132778 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:35.132749    2569 generic.go:356] "Generic (PLEG): container finished" podID="88eea35c-c9d1-4c86-8289-9fba5f31f6fd" containerID="ae188fab3ac08bf6b79622ac5d84a57ab1d8d3874f6b51156e55ce58f60f7e83" exitCode=0
Jul 12 17:05:35.132943 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:35.132834    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"88eea35c-c9d1-4c86-8289-9fba5f31f6fd","Type":"ContainerDied","Data":"ae188fab3ac08bf6b79622ac5d84a57ab1d8d3874f6b51156e55ce58f60f7e83"}
Jul 12 17:05:35.132943 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:35.132893    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"88eea35c-c9d1-4c86-8289-9fba5f31f6fd","Type":"ContainerStarted","Data":"a416eef9add41dad28a04ee3360dc0b9bcf1fe433517247207ab733607714309"}
Jul 12 17:05:36.139568 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:36.139533    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"88eea35c-c9d1-4c86-8289-9fba5f31f6fd","Type":"ContainerStarted","Data":"46c18f4c4c2a8235354bd566117a4ee191cc4903a703e779f90346e27cc4d560"}
Jul 12 17:05:36.139978 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:36.139597    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"88eea35c-c9d1-4c86-8289-9fba5f31f6fd","Type":"ContainerStarted","Data":"caf3b5104aae32a64afc50b6cd064c7c63c1b56c193fb376f91f2644e4980525"}
Jul 12 17:05:36.139978 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:36.139622    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"88eea35c-c9d1-4c86-8289-9fba5f31f6fd","Type":"ContainerStarted","Data":"210b8b1ec6ab7eb5846dd80825cefb0f508cb948930e004fb5628712e155aba4"}
Jul 12 17:05:36.139978 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:36.139646    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"88eea35c-c9d1-4c86-8289-9fba5f31f6fd","Type":"ContainerStarted","Data":"bfa492cd1adefb9a56639fbb49470016d426e8ea5ea449ca13f72983bd5fb5e5"}
Jul 12 17:05:36.139978 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:36.139676    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"88eea35c-c9d1-4c86-8289-9fba5f31f6fd","Type":"ContainerStarted","Data":"06d10c0f3a866e26da9cc7f4694a4f5e03fa37400132213a0922c27d09a5bd9b"}
Jul 12 17:05:36.139978 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:36.139700    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"88eea35c-c9d1-4c86-8289-9fba5f31f6fd","Type":"ContainerStarted","Data":"1e8775ee1125f7e3dc07a1c0aeeac4d373a577b1f68d6f9362adc52af0fc503c"}
Jul 12 17:05:36.167038 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:36.166995    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.149128853 podStartE2EDuration="2.166980597s" totalImagesPullingTime="0s" totalInitContainerRuntime="17.851744ms" isStatefulPod=false podCreationTimestamp="2026-07-12 17:05:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 17:05:36.165622169 +0000 UTC m=+144.285938967" watchObservedRunningTime="2026-07-12 17:05:36.166980597 +0000 UTC m=+144.287297424"
Jul 12 17:05:39.504001 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:05:39.503957    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 12 17:06:28.929017 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:06:28.928980    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8cd749d8c-2nznv"]
Jul 12 17:06:28.934173 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:06:28.934152    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8cd749d8c-2nznv"
Jul 12 17:06:28.937244 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:06:28.937218    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 12 17:06:28.938124 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:06:28.938103    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 12 17:06:28.938211 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:06:28.938198    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 12 17:06:28.938267 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:06:28.938209    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 12 17:06:28.938317 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:06:28.938291    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 12 17:06:28.938368 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:06:28.938345    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 12 17:06:28.939481 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:06:28.939464    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 12 17:06:28.946086 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:06:28.946065    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8cd749d8c-2nznv"]
Jul 12 17:06:29.057335 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:06:29.057307    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc0c4c7d-e7e4-4567-8cf8-d6a498f6439a-additional-service-ca\") pod \"cluster-proxy-proxy-agent-8cd749d8c-2nznv\" (UID: \"fc0c4c7d-e7e4-4567-8cf8-d6a498f6439a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8cd749d8c-2nznv"
Jul 12 17:06:29.057447 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:06:29.057348    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/fc0c4c7d-e7e4-4567-8cf8-d6a498f6439a-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-8cd749d8c-2nznv\" (UID: \"fc0c4c7d-e7e4-4567-8cf8-d6a498f6439a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8cd749d8c-2nznv"
Jul 12 17:06:29.057447 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:06:29.057366    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8hwc\" (UniqueName: \"kubernetes.io/projected/fc0c4c7d-e7e4-4567-8cf8-d6a498f6439a-kube-api-access-q8hwc\") pod \"cluster-proxy-proxy-agent-8cd749d8c-2nznv\" (UID: \"fc0c4c7d-e7e4-4567-8cf8-d6a498f6439a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8cd749d8c-2nznv"
Jul 12 17:06:29.057447 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:06:29.057429    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/fc0c4c7d-e7e4-4567-8cf8-d6a498f6439a-hub\") pod \"cluster-proxy-proxy-agent-8cd749d8c-2nznv\" (UID: \"fc0c4c7d-e7e4-4567-8cf8-d6a498f6439a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8cd749d8c-2nznv"
Jul 12 17:06:29.057575 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:06:29.057480    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fc0c4c7d-e7e4-4567-8cf8-d6a498f6439a-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-8cd749d8c-2nznv\" (UID: \"fc0c4c7d-e7e4-4567-8cf8-d6a498f6439a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8cd749d8c-2nznv"
Jul 12 17:06:29.057575 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:06:29.057501    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/fc0c4c7d-e7e4-4567-8cf8-d6a498f6439a-ca\") pod \"cluster-proxy-proxy-agent-8cd749d8c-2nznv\" (UID: \"fc0c4c7d-e7e4-4567-8cf8-d6a498f6439a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8cd749d8c-2nznv"
Jul 12 17:06:29.158081 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:06:29.158045    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fc0c4c7d-e7e4-4567-8cf8-d6a498f6439a-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-8cd749d8c-2nznv\" (UID: \"fc0c4c7d-e7e4-4567-8cf8-d6a498f6439a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8cd749d8c-2nznv"
Jul 12 17:06:29.158179 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:06:29.158088    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/fc0c4c7d-e7e4-4567-8cf8-d6a498f6439a-ca\") pod \"cluster-proxy-proxy-agent-8cd749d8c-2nznv\" (UID: \"fc0c4c7d-e7e4-4567-8cf8-d6a498f6439a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8cd749d8c-2nznv"
Jul 12 17:06:29.158179 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:06:29.158115    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc0c4c7d-e7e4-4567-8cf8-d6a498f6439a-additional-service-ca\") pod \"cluster-proxy-proxy-agent-8cd749d8c-2nznv\" (UID: \"fc0c4c7d-e7e4-4567-8cf8-d6a498f6439a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8cd749d8c-2nznv"
Jul 12 17:06:29.158179 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:06:29.158165    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/fc0c4c7d-e7e4-4567-8cf8-d6a498f6439a-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-8cd749d8c-2nznv\" (UID: \"fc0c4c7d-e7e4-4567-8cf8-d6a498f6439a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8cd749d8c-2nznv"
Jul 12 17:06:29.158315 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:06:29.158187    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8hwc\" (UniqueName: \"kubernetes.io/projected/fc0c4c7d-e7e4-4567-8cf8-d6a498f6439a-kube-api-access-q8hwc\") pod \"cluster-proxy-proxy-agent-8cd749d8c-2nznv\" (UID: \"fc0c4c7d-e7e4-4567-8cf8-d6a498f6439a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8cd749d8c-2nznv"
Jul 12 17:06:29.158315 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:06:29.158227    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/fc0c4c7d-e7e4-4567-8cf8-d6a498f6439a-hub\") pod \"cluster-proxy-proxy-agent-8cd749d8c-2nznv\" (UID: \"fc0c4c7d-e7e4-4567-8cf8-d6a498f6439a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8cd749d8c-2nznv"
Jul 12 17:06:29.159014 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:06:29.158987    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc0c4c7d-e7e4-4567-8cf8-d6a498f6439a-additional-service-ca\") pod \"cluster-proxy-proxy-agent-8cd749d8c-2nznv\" (UID: \"fc0c4c7d-e7e4-4567-8cf8-d6a498f6439a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8cd749d8c-2nznv"
Jul 12 17:06:29.160537 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:06:29.160497    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/fc0c4c7d-e7e4-4567-8cf8-d6a498f6439a-hub\") pod \"cluster-proxy-proxy-agent-8cd749d8c-2nznv\" (UID: \"fc0c4c7d-e7e4-4567-8cf8-d6a498f6439a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8cd749d8c-2nznv"
Jul 12 17:06:29.160628 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:06:29.160573    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/fc0c4c7d-e7e4-4567-8cf8-d6a498f6439a-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-8cd749d8c-2nznv\" (UID: \"fc0c4c7d-e7e4-4567-8cf8-d6a498f6439a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8cd749d8c-2nznv"
Jul 12 17:06:29.160628 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:06:29.160592    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/fc0c4c7d-e7e4-4567-8cf8-d6a498f6439a-ca\") pod \"cluster-proxy-proxy-agent-8cd749d8c-2nznv\" (UID: \"fc0c4c7d-e7e4-4567-8cf8-d6a498f6439a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8cd749d8c-2nznv"
Jul 12 17:06:29.160863 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:06:29.160841    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fc0c4c7d-e7e4-4567-8cf8-d6a498f6439a-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-8cd749d8c-2nznv\" (UID: \"fc0c4c7d-e7e4-4567-8cf8-d6a498f6439a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8cd749d8c-2nznv"
Jul 12 17:06:29.165963 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:06:29.165942    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8hwc\" (UniqueName: \"kubernetes.io/projected/fc0c4c7d-e7e4-4567-8cf8-d6a498f6439a-kube-api-access-q8hwc\") pod \"cluster-proxy-proxy-agent-8cd749d8c-2nznv\" (UID: \"fc0c4c7d-e7e4-4567-8cf8-d6a498f6439a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8cd749d8c-2nznv"
Jul 12 17:06:29.253116 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:06:29.253064    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8cd749d8c-2nznv"
Jul 12 17:06:29.378347 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:06:29.378315    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8cd749d8c-2nznv"]
Jul 12 17:06:29.381299 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:06:29.381270    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfc0c4c7d_e7e4_4567_8cf8_d6a498f6439a.slice/crio-f660f680951d71fde0ad4c54e0b0d16aab5353edfcbed9b0c6161015242966f7 WatchSource:0}: Error finding container f660f680951d71fde0ad4c54e0b0d16aab5353edfcbed9b0c6161015242966f7: Status 404 returned error can't find the container with id f660f680951d71fde0ad4c54e0b0d16aab5353edfcbed9b0c6161015242966f7
Jul 12 17:06:30.341113 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:06:30.340782    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8cd749d8c-2nznv" event={"ID":"fc0c4c7d-e7e4-4567-8cf8-d6a498f6439a","Type":"ContainerStarted","Data":"f660f680951d71fde0ad4c54e0b0d16aab5353edfcbed9b0c6161015242966f7"}
Jul 12 17:06:33.354834 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:06:33.354795    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8cd749d8c-2nznv" event={"ID":"fc0c4c7d-e7e4-4567-8cf8-d6a498f6439a","Type":"ContainerStarted","Data":"b7c546f378fca30d2e5e3b9b5499873ae931a18170888a4aba30531b8bc450bb"}
Jul 12 17:06:33.355189 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:06:33.354849    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8cd749d8c-2nznv" event={"ID":"fc0c4c7d-e7e4-4567-8cf8-d6a498f6439a","Type":"ContainerStarted","Data":"b8d9989b77a53a39b0e110b84b781c74645362c4e674a17b812766ed06cd4bd4"}
Jul 12 17:06:33.355189 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:06:33.354868    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8cd749d8c-2nznv" event={"ID":"fc0c4c7d-e7e4-4567-8cf8-d6a498f6439a","Type":"ContainerStarted","Data":"a7049f91a1eb8c12c713896c9f1f641b36cbb7fd35d055cbccaf449494b03a55"}
Jul 12 17:06:33.374835 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:06:33.374790    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8cd749d8c-2nznv" podStartSLOduration=1.961301399 podStartE2EDuration="5.374774913s" totalImagesPullingTime="3.413473514s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 17:06:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 17:06:33.372980967 +0000 UTC m=+201.493297766" watchObservedRunningTime="2026-07-12 17:06:33.374774913 +0000 UTC m=+201.495091716"
Jul 12 17:06:34.504523 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:06:34.504458    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 12 17:06:34.519698 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:06:34.519669    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 12 17:06:35.377928 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:06:35.377896    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 12 17:07:34.311792 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:07:34.311718    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-6459574568-7rcnn"]
Jul 12 17:07:34.313787 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:07:34.313770    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6459574568-7rcnn"
Jul 12 17:07:34.316086 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:07:34.316063    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 12 17:07:34.316086 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:07:34.316063    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-jjvqt\""
Jul 12 17:07:34.316962 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:07:34.316921    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 12 17:07:34.316962 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:07:34.316927    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 12 17:07:34.317168 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:07:34.316971    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 12 17:07:34.317168 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:07:34.317153    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 12 17:07:34.328049 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:07:34.328029    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6459574568-7rcnn"]
Jul 12 17:07:34.354628 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:07:34.354600    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/bccd8821-f075-42d5-88a0-28530ae56bac-metrics-cert\") pod \"lws-controller-manager-6459574568-7rcnn\" (UID: \"bccd8821-f075-42d5-88a0-28530ae56bac\") " pod="openshift-lws-operator/lws-controller-manager-6459574568-7rcnn"
Jul 12 17:07:34.354725 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:07:34.354637    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/bccd8821-f075-42d5-88a0-28530ae56bac-manager-config\") pod \"lws-controller-manager-6459574568-7rcnn\" (UID: \"bccd8821-f075-42d5-88a0-28530ae56bac\") " pod="openshift-lws-operator/lws-controller-manager-6459574568-7rcnn"
Jul 12 17:07:34.354725 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:07:34.354666    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bccd8821-f075-42d5-88a0-28530ae56bac-cert\") pod \"lws-controller-manager-6459574568-7rcnn\" (UID: \"bccd8821-f075-42d5-88a0-28530ae56bac\") " pod="openshift-lws-operator/lws-controller-manager-6459574568-7rcnn"
Jul 12 17:07:34.354811 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:07:34.354752    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zj8h\" (UniqueName: \"kubernetes.io/projected/bccd8821-f075-42d5-88a0-28530ae56bac-kube-api-access-5zj8h\") pod \"lws-controller-manager-6459574568-7rcnn\" (UID: \"bccd8821-f075-42d5-88a0-28530ae56bac\") " pod="openshift-lws-operator/lws-controller-manager-6459574568-7rcnn"
Jul 12 17:07:34.455895 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:07:34.455865    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zj8h\" (UniqueName: \"kubernetes.io/projected/bccd8821-f075-42d5-88a0-28530ae56bac-kube-api-access-5zj8h\") pod \"lws-controller-manager-6459574568-7rcnn\" (UID: \"bccd8821-f075-42d5-88a0-28530ae56bac\") " pod="openshift-lws-operator/lws-controller-manager-6459574568-7rcnn"
Jul 12 17:07:34.456017 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:07:34.455923    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/bccd8821-f075-42d5-88a0-28530ae56bac-metrics-cert\") pod \"lws-controller-manager-6459574568-7rcnn\" (UID: \"bccd8821-f075-42d5-88a0-28530ae56bac\") " pod="openshift-lws-operator/lws-controller-manager-6459574568-7rcnn"
Jul 12 17:07:34.456017 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:07:34.455951    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/bccd8821-f075-42d5-88a0-28530ae56bac-manager-config\") pod \"lws-controller-manager-6459574568-7rcnn\" (UID: \"bccd8821-f075-42d5-88a0-28530ae56bac\") " pod="openshift-lws-operator/lws-controller-manager-6459574568-7rcnn"
Jul 12 17:07:34.456017 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:07:34.455981    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bccd8821-f075-42d5-88a0-28530ae56bac-cert\") pod \"lws-controller-manager-6459574568-7rcnn\" (UID: \"bccd8821-f075-42d5-88a0-28530ae56bac\") " pod="openshift-lws-operator/lws-controller-manager-6459574568-7rcnn"
Jul 12 17:07:34.456700 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:07:34.456680    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/bccd8821-f075-42d5-88a0-28530ae56bac-manager-config\") pod \"lws-controller-manager-6459574568-7rcnn\" (UID: \"bccd8821-f075-42d5-88a0-28530ae56bac\") " pod="openshift-lws-operator/lws-controller-manager-6459574568-7rcnn"
Jul 12 17:07:34.458357 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:07:34.458335    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bccd8821-f075-42d5-88a0-28530ae56bac-cert\") pod \"lws-controller-manager-6459574568-7rcnn\" (UID: \"bccd8821-f075-42d5-88a0-28530ae56bac\") " pod="openshift-lws-operator/lws-controller-manager-6459574568-7rcnn"
Jul 12 17:07:34.458429 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:07:34.458408    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/bccd8821-f075-42d5-88a0-28530ae56bac-metrics-cert\") pod \"lws-controller-manager-6459574568-7rcnn\" (UID: \"bccd8821-f075-42d5-88a0-28530ae56bac\") " pod="openshift-lws-operator/lws-controller-manager-6459574568-7rcnn"
Jul 12 17:07:34.463954 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:07:34.463929    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zj8h\" (UniqueName: \"kubernetes.io/projected/bccd8821-f075-42d5-88a0-28530ae56bac-kube-api-access-5zj8h\") pod \"lws-controller-manager-6459574568-7rcnn\" (UID: \"bccd8821-f075-42d5-88a0-28530ae56bac\") " pod="openshift-lws-operator/lws-controller-manager-6459574568-7rcnn"
Jul 12 17:07:34.623218 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:07:34.623144    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6459574568-7rcnn"
Jul 12 17:07:34.746066 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:07:34.746027    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6459574568-7rcnn"]
Jul 12 17:07:34.748824 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:07:34.748790    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbccd8821_f075_42d5_88a0_28530ae56bac.slice/crio-616da11db7ae07c8278dfc159e691ff4de72d5ac4936d8a8cb50bba73ebeb3d0 WatchSource:0}: Error finding container 616da11db7ae07c8278dfc159e691ff4de72d5ac4936d8a8cb50bba73ebeb3d0: Status 404 returned error can't find the container with id 616da11db7ae07c8278dfc159e691ff4de72d5ac4936d8a8cb50bba73ebeb3d0
Jul 12 17:07:35.575257 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:07:35.575224    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6459574568-7rcnn" event={"ID":"bccd8821-f075-42d5-88a0-28530ae56bac","Type":"ContainerStarted","Data":"616da11db7ae07c8278dfc159e691ff4de72d5ac4936d8a8cb50bba73ebeb3d0"}
Jul 12 17:07:38.587351 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:07:38.587312    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6459574568-7rcnn" event={"ID":"bccd8821-f075-42d5-88a0-28530ae56bac","Type":"ContainerStarted","Data":"cd6f9cb80569eee99fb34eeb3542e2ece0c22a4217a9e59105f322adc3dd0881"}
Jul 12 17:07:38.587803 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:07:38.587436    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-6459574568-7rcnn"
Jul 12 17:07:38.619011 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:07:38.618964    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-6459574568-7rcnn" podStartSLOduration=1.810262716 podStartE2EDuration="4.618952236s" totalImagesPullingTime="2.80868952s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 17:07:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 17:07:38.617880917 +0000 UTC m=+266.738197718" watchObservedRunningTime="2026-07-12 17:07:38.618952236 +0000 UTC m=+266.739269032"
Jul 12 17:07:38.632826 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:07:38.632778    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-554bd9d4cc-b2lhx"]
Jul 12 17:07:38.636119 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:07:38.636101    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-554bd9d4cc-b2lhx"
Jul 12 17:07:38.638273 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:07:38.638254    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 12 17:07:38.638433 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:07:38.638415    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 12 17:07:38.638546 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:07:38.638532    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 12 17:07:38.638878 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:07:38.638862    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 12 17:07:38.638934 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:07:38.638919    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-6fnfz\""
Jul 12 17:07:38.655940 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:07:38.655910    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-554bd9d4cc-b2lhx"]
Jul 12 17:07:38.686299 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:07:38.686273    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1aa36e66-d3fd-4083-b35a-e62fefcacdd9-manifests\") pod \"opendatahub-operator-controller-manager-554bd9d4cc-b2lhx\" (UID: \"1aa36e66-d3fd-4083-b35a-e62fefcacdd9\") " pod="opendatahub/opendatahub-operator-controller-manager-554bd9d4cc-b2lhx"
Jul 12 17:07:38.686405 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:07:38.686326    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1aa36e66-d3fd-4083-b35a-e62fefcacdd9-webhook-cert\") pod \"opendatahub-operator-controller-manager-554bd9d4cc-b2lhx\" (UID: \"1aa36e66-d3fd-4083-b35a-e62fefcacdd9\") " pod="opendatahub/opendatahub-operator-controller-manager-554bd9d4cc-b2lhx"
Jul 12 17:07:38.686405 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:07:38.686352    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6fd6\" (UniqueName: \"kubernetes.io/projected/1aa36e66-d3fd-4083-b35a-e62fefcacdd9-kube-api-access-b6fd6\") pod \"opendatahub-operator-controller-manager-554bd9d4cc-b2lhx\" (UID: \"1aa36e66-d3fd-4083-b35a-e62fefcacdd9\") " pod="opendatahub/opendatahub-operator-controller-manager-554bd9d4cc-b2lhx"
Jul 12 17:07:38.686479 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:07:38.686433    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1aa36e66-d3fd-4083-b35a-e62fefcacdd9-tmp\") pod \"opendatahub-operator-controller-manager-554bd9d4cc-b2lhx\" (UID: \"1aa36e66-d3fd-4083-b35a-e62fefcacdd9\") " pod="opendatahub/opendatahub-operator-controller-manager-554bd9d4cc-b2lhx"
Jul 12 17:07:38.686590 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:07:38.686569    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1aa36e66-d3fd-4083-b35a-e62fefcacdd9-apiservice-cert\") pod \"opendatahub-operator-controller-manager-554bd9d4cc-b2lhx\" (UID: \"1aa36e66-d3fd-4083-b35a-e62fefcacdd9\") " pod="opendatahub/opendatahub-operator-controller-manager-554bd9d4cc-b2lhx"
Jul 12 17:07:38.787177 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:07:38.787150    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1aa36e66-d3fd-4083-b35a-e62fefcacdd9-tmp\") pod \"opendatahub-operator-controller-manager-554bd9d4cc-b2lhx\" (UID: \"1aa36e66-d3fd-4083-b35a-e62fefcacdd9\") " pod="opendatahub/opendatahub-operator-controller-manager-554bd9d4cc-b2lhx"
Jul 12 17:07:38.787283 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:07:38.787196    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1aa36e66-d3fd-4083-b35a-e62fefcacdd9-apiservice-cert\") pod \"opendatahub-operator-controller-manager-554bd9d4cc-b2lhx\" (UID: \"1aa36e66-d3fd-4083-b35a-e62fefcacdd9\") " pod="opendatahub/opendatahub-operator-controller-manager-554bd9d4cc-b2lhx"
Jul 12 17:07:38.787283 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:07:38.787243    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1aa36e66-d3fd-4083-b35a-e62fefcacdd9-manifests\") pod \"opendatahub-operator-controller-manager-554bd9d4cc-b2lhx\" (UID: \"1aa36e66-d3fd-4083-b35a-e62fefcacdd9\") " pod="opendatahub/opendatahub-operator-controller-manager-554bd9d4cc-b2lhx"
Jul 12 17:07:38.787394 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:07:38.787284    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1aa36e66-d3fd-4083-b35a-e62fefcacdd9-webhook-cert\") pod \"opendatahub-operator-controller-manager-554bd9d4cc-b2lhx\" (UID: \"1aa36e66-d3fd-4083-b35a-e62fefcacdd9\") " pod="opendatahub/opendatahub-operator-controller-manager-554bd9d4cc-b2lhx"
Jul 12 17:07:38.787394 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:07:38.787311    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6fd6\" (UniqueName: \"kubernetes.io/projected/1aa36e66-d3fd-4083-b35a-e62fefcacdd9-kube-api-access-b6fd6\") pod \"opendatahub-operator-controller-manager-554bd9d4cc-b2lhx\" (UID: \"1aa36e66-d3fd-4083-b35a-e62fefcacdd9\") " pod="opendatahub/opendatahub-operator-controller-manager-554bd9d4cc-b2lhx"
Jul 12 17:07:38.787707 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:07:38.787678    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1aa36e66-d3fd-4083-b35a-e62fefcacdd9-manifests\") pod \"opendatahub-operator-controller-manager-554bd9d4cc-b2lhx\" (UID: \"1aa36e66-d3fd-4083-b35a-e62fefcacdd9\") " pod="opendatahub/opendatahub-operator-controller-manager-554bd9d4cc-b2lhx"
Jul 12 17:07:38.789541 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:07:38.789496    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1aa36e66-d3fd-4083-b35a-e62fefcacdd9-tmp\") pod \"opendatahub-operator-controller-manager-554bd9d4cc-b2lhx\" (UID: \"1aa36e66-d3fd-4083-b35a-e62fefcacdd9\") " pod="opendatahub/opendatahub-operator-controller-manager-554bd9d4cc-b2lhx"
Jul 12 17:07:38.789629 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:07:38.789594    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1aa36e66-d3fd-4083-b35a-e62fefcacdd9-webhook-cert\") pod \"opendatahub-operator-controller-manager-554bd9d4cc-b2lhx\" (UID: \"1aa36e66-d3fd-4083-b35a-e62fefcacdd9\") " pod="opendatahub/opendatahub-operator-controller-manager-554bd9d4cc-b2lhx"
Jul 12 17:07:38.789770 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:07:38.789749    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1aa36e66-d3fd-4083-b35a-e62fefcacdd9-apiservice-cert\") pod \"opendatahub-operator-controller-manager-554bd9d4cc-b2lhx\" (UID: \"1aa36e66-d3fd-4083-b35a-e62fefcacdd9\") " pod="opendatahub/opendatahub-operator-controller-manager-554bd9d4cc-b2lhx"
Jul 12 17:07:38.796219 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:07:38.796184    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6fd6\" (UniqueName: \"kubernetes.io/projected/1aa36e66-d3fd-4083-b35a-e62fefcacdd9-kube-api-access-b6fd6\") pod \"opendatahub-operator-controller-manager-554bd9d4cc-b2lhx\" (UID: \"1aa36e66-d3fd-4083-b35a-e62fefcacdd9\") " pod="opendatahub/opendatahub-operator-controller-manager-554bd9d4cc-b2lhx"
Jul 12 17:07:38.946219 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:07:38.946196    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-554bd9d4cc-b2lhx"
Jul 12 17:07:39.076624 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:07:39.076596    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-554bd9d4cc-b2lhx"]
Jul 12 17:07:39.079339 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:07:39.079308    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1aa36e66_d3fd_4083_b35a_e62fefcacdd9.slice/crio-6128eec382902da6c26be62f3e3fae49ddbf81c9747b3fcaf5d616f2ae8a0fc9 WatchSource:0}: Error finding container 6128eec382902da6c26be62f3e3fae49ddbf81c9747b3fcaf5d616f2ae8a0fc9: Status 404 returned error can't find the container with id 6128eec382902da6c26be62f3e3fae49ddbf81c9747b3fcaf5d616f2ae8a0fc9
Jul 12 17:07:39.591859 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:07:39.591826    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-554bd9d4cc-b2lhx" event={"ID":"1aa36e66-d3fd-4083-b35a-e62fefcacdd9","Type":"ContainerStarted","Data":"6128eec382902da6c26be62f3e3fae49ddbf81c9747b3fcaf5d616f2ae8a0fc9"}
Jul 12 17:07:42.605719 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:07:42.605687    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-554bd9d4cc-b2lhx" event={"ID":"1aa36e66-d3fd-4083-b35a-e62fefcacdd9","Type":"ContainerStarted","Data":"d152d025fee87fdbf456f56e258eccab363994c97582c7c470c31441298f7a73"}
Jul 12 17:07:43.610440 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:07:43.610402    2569 generic.go:356] "Generic (PLEG): container finished" podID="1aa36e66-d3fd-4083-b35a-e62fefcacdd9" containerID="d152d025fee87fdbf456f56e258eccab363994c97582c7c470c31441298f7a73" exitCode=0
Jul 12 17:07:43.610798 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:07:43.610471    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-554bd9d4cc-b2lhx" event={"ID":"1aa36e66-d3fd-4083-b35a-e62fefcacdd9","Type":"ContainerDied","Data":"d152d025fee87fdbf456f56e258eccab363994c97582c7c470c31441298f7a73"}
Jul 12 17:07:44.615857 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:07:44.615822    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-554bd9d4cc-b2lhx" event={"ID":"1aa36e66-d3fd-4083-b35a-e62fefcacdd9","Type":"ContainerStarted","Data":"62b4d3e29c6f6b90e243be36f100711507e6b74367b2662a09ad02a65d7f3604"}
Jul 12 17:07:44.616235 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:07:44.615969    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-554bd9d4cc-b2lhx"
Jul 12 17:07:44.636362 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:07:44.636314    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-554bd9d4cc-b2lhx" podStartSLOduration=2.313701396 podStartE2EDuration="6.636295204s" totalImagesPullingTime="3.233445518s" totalInitContainerRuntime="1.08914829s" isStatefulPod=false podCreationTimestamp="2026-07-12 17:07:38 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 17:07:44.634823546 +0000 UTC m=+272.755140343" watchObservedRunningTime="2026-07-12 17:07:44.636295204 +0000 UTC m=+272.756612001"
Jul 12 17:07:49.594079 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:07:49.594046    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-6459574568-7rcnn"
Jul 12 17:07:55.621958 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:07:55.621926    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-554bd9d4cc-b2lhx"
Jul 12 17:07:58.074456 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:07:58.074421    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-59584446bf-bxv2x"]
Jul 12 17:07:58.086124 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:07:58.086104    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-59584446bf-bxv2x"
Jul 12 17:07:58.089554 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:07:58.089531    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 12 17:07:58.089893 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:07:58.089874    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Jul 12 17:07:58.090003 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:07:58.089879    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 12 17:07:58.090583 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:07:58.090517    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-dfm9d\""
Jul 12 17:07:58.090583 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:07:58.090503    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 12 17:07:58.096624 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:07:58.096597    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-59584446bf-bxv2x"]
Jul 12 17:07:58.241983 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:07:58.241940    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26zfw\" (UniqueName: \"kubernetes.io/projected/327a7007-ca31-49a8-acda-91bf35ed6963-kube-api-access-26zfw\") pod \"kube-auth-proxy-59584446bf-bxv2x\" (UID: \"327a7007-ca31-49a8-acda-91bf35ed6963\") " pod="openshift-ingress/kube-auth-proxy-59584446bf-bxv2x"
Jul 12 17:07:58.242160 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:07:58.241997    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/327a7007-ca31-49a8-acda-91bf35ed6963-tls-certs\") pod \"kube-auth-proxy-59584446bf-bxv2x\" (UID: \"327a7007-ca31-49a8-acda-91bf35ed6963\") " pod="openshift-ingress/kube-auth-proxy-59584446bf-bxv2x"
Jul 12 17:07:58.242160 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:07:58.242045    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/327a7007-ca31-49a8-acda-91bf35ed6963-tmp\") pod \"kube-auth-proxy-59584446bf-bxv2x\" (UID: \"327a7007-ca31-49a8-acda-91bf35ed6963\") " pod="openshift-ingress/kube-auth-proxy-59584446bf-bxv2x"
Jul 12 17:07:58.343152 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:07:58.343072    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/327a7007-ca31-49a8-acda-91bf35ed6963-tls-certs\") pod \"kube-auth-proxy-59584446bf-bxv2x\" (UID: \"327a7007-ca31-49a8-acda-91bf35ed6963\") " pod="openshift-ingress/kube-auth-proxy-59584446bf-bxv2x"
Jul 12 17:07:58.343152 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:07:58.343126    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/327a7007-ca31-49a8-acda-91bf35ed6963-tmp\") pod \"kube-auth-proxy-59584446bf-bxv2x\" (UID: \"327a7007-ca31-49a8-acda-91bf35ed6963\") " pod="openshift-ingress/kube-auth-proxy-59584446bf-bxv2x"
Jul 12 17:07:58.343361 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:07:58.343254    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26zfw\" (UniqueName: \"kubernetes.io/projected/327a7007-ca31-49a8-acda-91bf35ed6963-kube-api-access-26zfw\") pod \"kube-auth-proxy-59584446bf-bxv2x\" (UID: \"327a7007-ca31-49a8-acda-91bf35ed6963\") " pod="openshift-ingress/kube-auth-proxy-59584446bf-bxv2x"
Jul 12 17:07:58.345326 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:07:58.345288    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/327a7007-ca31-49a8-acda-91bf35ed6963-tmp\") pod \"kube-auth-proxy-59584446bf-bxv2x\" (UID: \"327a7007-ca31-49a8-acda-91bf35ed6963\") " pod="openshift-ingress/kube-auth-proxy-59584446bf-bxv2x"
Jul 12 17:07:58.345453 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:07:58.345432    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/327a7007-ca31-49a8-acda-91bf35ed6963-tls-certs\") pod \"kube-auth-proxy-59584446bf-bxv2x\" (UID: \"327a7007-ca31-49a8-acda-91bf35ed6963\") " pod="openshift-ingress/kube-auth-proxy-59584446bf-bxv2x"
Jul 12 17:07:58.352601 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:07:58.352578    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-26zfw\" (UniqueName: \"kubernetes.io/projected/327a7007-ca31-49a8-acda-91bf35ed6963-kube-api-access-26zfw\") pod \"kube-auth-proxy-59584446bf-bxv2x\" (UID: \"327a7007-ca31-49a8-acda-91bf35ed6963\") " pod="openshift-ingress/kube-auth-proxy-59584446bf-bxv2x"
Jul 12 17:07:58.396367 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:07:58.396335    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-59584446bf-bxv2x"
Jul 12 17:07:58.524176 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:07:58.524030    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-59584446bf-bxv2x"]
Jul 12 17:07:58.526744 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:07:58.526717    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod327a7007_ca31_49a8_acda_91bf35ed6963.slice/crio-afeefe039a52d38424ff12441f03d6ba35491b75b260444c44a405046aef87b7 WatchSource:0}: Error finding container afeefe039a52d38424ff12441f03d6ba35491b75b260444c44a405046aef87b7: Status 404 returned error can't find the container with id afeefe039a52d38424ff12441f03d6ba35491b75b260444c44a405046aef87b7
Jul 12 17:07:58.672974 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:07:58.672942    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-59584446bf-bxv2x" event={"ID":"327a7007-ca31-49a8-acda-91bf35ed6963","Type":"ContainerStarted","Data":"afeefe039a52d38424ff12441f03d6ba35491b75b260444c44a405046aef87b7"}
Jul 12 17:08:01.687739 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:08:01.687703    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-59584446bf-bxv2x" event={"ID":"327a7007-ca31-49a8-acda-91bf35ed6963","Type":"ContainerStarted","Data":"04866ba091e1a7c08a6b5518c6728b39b263a2023af3876fa4b9b53e450bfe41"}
Jul 12 17:08:01.702806 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:08:01.702762    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-59584446bf-bxv2x" podStartSLOduration=0.948905023 podStartE2EDuration="3.702748344s" totalImagesPullingTime="2.753843321s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 17:07:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 17:08:01.702145012 +0000 UTC m=+289.822461809" watchObservedRunningTime="2026-07-12 17:08:01.702748344 +0000 UTC m=+289.823065142"
Jul 12 17:08:12.249005 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:08:12.248965    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9xbrj_6b618491-4e81-4708-ab1b-2781d1df7c87/ovn-acl-logging/0.log"
Jul 12 17:08:12.253613 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:08:12.253594    2569 kubelet.go:1648] "Image garbage collection succeeded"
Jul 12 17:08:12.262005 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:08:12.261987    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9xbrj_6b618491-4e81-4708-ab1b-2781d1df7c87/ovn-acl-logging/0.log"
Jul 12 17:09:40.463189 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:09:40.463155    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-cmbk5"]
Jul 12 17:09:40.466176 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:09:40.466159    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-cmbk5"
Jul 12 17:09:40.468613 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:09:40.468589    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 12 17:09:40.469208 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:09:40.469188    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Jul 12 17:09:40.469315 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:09:40.469189    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Jul 12 17:09:40.469315 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:09:40.469221    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 12 17:09:40.469315 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:09:40.469227    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-2z6gm\""
Jul 12 17:09:40.474373 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:09:40.474352    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-cmbk5"]
Jul 12 17:09:40.606598 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:09:40.606567    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/9b68d9a9-6bc7-4f78-a3dd-b5a56a5aa7d4-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-cmbk5\" (UID: \"9b68d9a9-6bc7-4f78-a3dd-b5a56a5aa7d4\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-cmbk5"
Jul 12 17:09:40.606765 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:09:40.606657    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/9b68d9a9-6bc7-4f78-a3dd-b5a56a5aa7d4-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-cmbk5\" (UID: \"9b68d9a9-6bc7-4f78-a3dd-b5a56a5aa7d4\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-cmbk5"
Jul 12 17:09:40.606765 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:09:40.606716    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qlfj\" (UniqueName: \"kubernetes.io/projected/9b68d9a9-6bc7-4f78-a3dd-b5a56a5aa7d4-kube-api-access-9qlfj\") pod \"kuadrant-console-plugin-66b5774985-cmbk5\" (UID: \"9b68d9a9-6bc7-4f78-a3dd-b5a56a5aa7d4\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-cmbk5"
Jul 12 17:09:40.708091 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:09:40.708048    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/9b68d9a9-6bc7-4f78-a3dd-b5a56a5aa7d4-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-cmbk5\" (UID: \"9b68d9a9-6bc7-4f78-a3dd-b5a56a5aa7d4\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-cmbk5"
Jul 12 17:09:40.708288 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:09:40.708118    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qlfj\" (UniqueName: \"kubernetes.io/projected/9b68d9a9-6bc7-4f78-a3dd-b5a56a5aa7d4-kube-api-access-9qlfj\") pod \"kuadrant-console-plugin-66b5774985-cmbk5\" (UID: \"9b68d9a9-6bc7-4f78-a3dd-b5a56a5aa7d4\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-cmbk5"
Jul 12 17:09:40.708288 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:09:40.708185    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/9b68d9a9-6bc7-4f78-a3dd-b5a56a5aa7d4-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-cmbk5\" (UID: \"9b68d9a9-6bc7-4f78-a3dd-b5a56a5aa7d4\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-cmbk5"
Jul 12 17:09:40.709402 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:09:40.709368    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/9b68d9a9-6bc7-4f78-a3dd-b5a56a5aa7d4-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-cmbk5\" (UID: \"9b68d9a9-6bc7-4f78-a3dd-b5a56a5aa7d4\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-cmbk5"
Jul 12 17:09:40.712042 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:09:40.712010    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/9b68d9a9-6bc7-4f78-a3dd-b5a56a5aa7d4-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-cmbk5\" (UID: \"9b68d9a9-6bc7-4f78-a3dd-b5a56a5aa7d4\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-cmbk5"
Jul 12 17:09:40.716152 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:09:40.716084    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qlfj\" (UniqueName: \"kubernetes.io/projected/9b68d9a9-6bc7-4f78-a3dd-b5a56a5aa7d4-kube-api-access-9qlfj\") pod \"kuadrant-console-plugin-66b5774985-cmbk5\" (UID: \"9b68d9a9-6bc7-4f78-a3dd-b5a56a5aa7d4\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-cmbk5"
Jul 12 17:09:40.775917 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:09:40.775871    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-cmbk5"
Jul 12 17:09:40.898150 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:09:40.898125    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-cmbk5"]
Jul 12 17:09:40.900490 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:09:40.900465    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9b68d9a9_6bc7_4f78_a3dd_b5a56a5aa7d4.slice/crio-b70200403874ee2b3fa0e68286148cff4942b3a3c439ffecc2f31fff847c8f25 WatchSource:0}: Error finding container b70200403874ee2b3fa0e68286148cff4942b3a3c439ffecc2f31fff847c8f25: Status 404 returned error can't find the container with id b70200403874ee2b3fa0e68286148cff4942b3a3c439ffecc2f31fff847c8f25
Jul 12 17:09:40.901862 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:09:40.901841    2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 12 17:09:41.074263 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:09:41.074197    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-cmbk5" event={"ID":"9b68d9a9-6bc7-4f78-a3dd-b5a56a5aa7d4","Type":"ContainerStarted","Data":"b70200403874ee2b3fa0e68286148cff4942b3a3c439ffecc2f31fff847c8f25"}
Jul 12 17:10:02.175430 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:10:02.175391    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-cmbk5" event={"ID":"9b68d9a9-6bc7-4f78-a3dd-b5a56a5aa7d4","Type":"ContainerStarted","Data":"9891ec254a8d909f3034819c694da7a1fca297e18c7ba94b3c075a5b8087414d"}
Jul 12 17:10:02.193206 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:10:02.193164    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-cmbk5" podStartSLOduration=1.434363627 podStartE2EDuration="22.193150686s" totalImagesPullingTime="20.758787059s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 17:09:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 17:10:02.192071058 +0000 UTC m=+410.312387856" watchObservedRunningTime="2026-07-12 17:10:02.193150686 +0000 UTC m=+410.313467484"
Jul 12 17:10:34.030271 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:10:34.030200    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl4zn"]
Jul 12 17:10:34.086354 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:10:34.086326    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl4zn"]
Jul 12 17:10:34.086354 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:10:34.086360    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl4zn"]
Jul 12 17:10:34.086554 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:10:34.086438    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-hl4zn"
Jul 12 17:10:34.088618 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:10:34.088597    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 12 17:10:34.142723 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:10:34.142693    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8d97l\" (UniqueName: \"kubernetes.io/projected/3715331f-8a45-4780-9bfe-ceae9bea46ec-kube-api-access-8d97l\") pod \"limitador-limitador-54d6d988d5-hl4zn\" (UID: \"3715331f-8a45-4780-9bfe-ceae9bea46ec\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-hl4zn"
Jul 12 17:10:34.142831 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:10:34.142772    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3715331f-8a45-4780-9bfe-ceae9bea46ec-config-file\") pod \"limitador-limitador-54d6d988d5-hl4zn\" (UID: \"3715331f-8a45-4780-9bfe-ceae9bea46ec\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-hl4zn"
Jul 12 17:10:34.243668 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:10:34.243641    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3715331f-8a45-4780-9bfe-ceae9bea46ec-config-file\") pod \"limitador-limitador-54d6d988d5-hl4zn\" (UID: \"3715331f-8a45-4780-9bfe-ceae9bea46ec\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-hl4zn"
Jul 12 17:10:34.243762 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:10:34.243681    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8d97l\" (UniqueName: \"kubernetes.io/projected/3715331f-8a45-4780-9bfe-ceae9bea46ec-kube-api-access-8d97l\") pod \"limitador-limitador-54d6d988d5-hl4zn\" (UID: \"3715331f-8a45-4780-9bfe-ceae9bea46ec\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-hl4zn"
Jul 12 17:10:34.244224 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:10:34.244206    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3715331f-8a45-4780-9bfe-ceae9bea46ec-config-file\") pod \"limitador-limitador-54d6d988d5-hl4zn\" (UID: \"3715331f-8a45-4780-9bfe-ceae9bea46ec\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-hl4zn"
Jul 12 17:10:34.251284 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:10:34.251263    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8d97l\" (UniqueName: \"kubernetes.io/projected/3715331f-8a45-4780-9bfe-ceae9bea46ec-kube-api-access-8d97l\") pod \"limitador-limitador-54d6d988d5-hl4zn\" (UID: \"3715331f-8a45-4780-9bfe-ceae9bea46ec\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-hl4zn"
Jul 12 17:10:34.396541 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:10:34.396471    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-hl4zn"
Jul 12 17:10:34.520273 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:10:34.520251    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl4zn"]
Jul 12 17:10:34.522738 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:10:34.522708    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3715331f_8a45_4780_9bfe_ceae9bea46ec.slice/crio-e45f5a790ddc52b1a216781e2443c5d1d47e8698b802c2c5a63400442afd307d WatchSource:0}: Error finding container e45f5a790ddc52b1a216781e2443c5d1d47e8698b802c2c5a63400442afd307d: Status 404 returned error can't find the container with id e45f5a790ddc52b1a216781e2443c5d1d47e8698b802c2c5a63400442afd307d
Jul 12 17:10:34.681650 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:10:34.681618    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-kfk5j"]
Jul 12 17:10:34.685801 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:10:34.685784    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-kfk5j"
Jul 12 17:10:34.688354 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:10:34.688330    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-j8ms5\""
Jul 12 17:10:34.689544 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:10:34.689503    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-kfk5j"]
Jul 12 17:10:34.747832 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:10:34.747805    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vhrv\" (UniqueName: \"kubernetes.io/projected/29eca76d-1ab5-43a5-a4da-af12c349cae0-kube-api-access-8vhrv\") pod \"authorino-75d556bd77-kfk5j\" (UID: \"29eca76d-1ab5-43a5-a4da-af12c349cae0\") " pod="kuadrant-system/authorino-75d556bd77-kfk5j"
Jul 12 17:10:34.848561 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:10:34.848530    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vhrv\" (UniqueName: \"kubernetes.io/projected/29eca76d-1ab5-43a5-a4da-af12c349cae0-kube-api-access-8vhrv\") pod \"authorino-75d556bd77-kfk5j\" (UID: \"29eca76d-1ab5-43a5-a4da-af12c349cae0\") " pod="kuadrant-system/authorino-75d556bd77-kfk5j"
Jul 12 17:10:34.856635 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:10:34.856603    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vhrv\" (UniqueName: \"kubernetes.io/projected/29eca76d-1ab5-43a5-a4da-af12c349cae0-kube-api-access-8vhrv\") pod \"authorino-75d556bd77-kfk5j\" (UID: \"29eca76d-1ab5-43a5-a4da-af12c349cae0\") " pod="kuadrant-system/authorino-75d556bd77-kfk5j"
Jul 12 17:10:34.995284 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:10:34.995191    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-kfk5j"
Jul 12 17:10:35.156157 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:10:35.156122    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-kfk5j"]
Jul 12 17:10:35.162112 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:10:35.162077    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod29eca76d_1ab5_43a5_a4da_af12c349cae0.slice/crio-89d1bd134c0a66d7bea21a9928bd3112e99bdabed2f477e87674a1c290ae4243 WatchSource:0}: Error finding container 89d1bd134c0a66d7bea21a9928bd3112e99bdabed2f477e87674a1c290ae4243: Status 404 returned error can't find the container with id 89d1bd134c0a66d7bea21a9928bd3112e99bdabed2f477e87674a1c290ae4243
Jul 12 17:10:35.313038 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:10:35.312938    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-kfk5j" event={"ID":"29eca76d-1ab5-43a5-a4da-af12c349cae0","Type":"ContainerStarted","Data":"89d1bd134c0a66d7bea21a9928bd3112e99bdabed2f477e87674a1c290ae4243"}
Jul 12 17:10:35.315070 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:10:35.314947    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-hl4zn" event={"ID":"3715331f-8a45-4780-9bfe-ceae9bea46ec","Type":"ContainerStarted","Data":"e45f5a790ddc52b1a216781e2443c5d1d47e8698b802c2c5a63400442afd307d"}
Jul 12 17:10:39.337071 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:10:39.337031    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-kfk5j" event={"ID":"29eca76d-1ab5-43a5-a4da-af12c349cae0","Type":"ContainerStarted","Data":"09515853e02f18ea15d1179c93749a87fb7670c6fe3029890a3b773316644d20"}
Jul 12 17:10:39.338481 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:10:39.338453    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-hl4zn" event={"ID":"3715331f-8a45-4780-9bfe-ceae9bea46ec","Type":"ContainerStarted","Data":"6d49121ed4482aa856ba705d1b8c0b9dd63baf48e70dfb012707dd1ef2481626"}
Jul 12 17:10:39.338609 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:10:39.338535    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-hl4zn"
Jul 12 17:10:39.354293 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:10:39.354249    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-kfk5j" podStartSLOduration=1.752269315 podStartE2EDuration="5.354232455s" totalImagesPullingTime="3.60196314s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 17:10:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 17:10:39.352706816 +0000 UTC m=+447.473023614" watchObservedRunningTime="2026-07-12 17:10:39.354232455 +0000 UTC m=+447.474549230"
Jul 12 17:10:39.368940 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:10:39.368892    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-hl4zn" podStartSLOduration=1.13281295 podStartE2EDuration="5.368880277s" totalImagesPullingTime="4.236067327s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 17:10:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 17:10:39.367887325 +0000 UTC m=+447.488204122" watchObservedRunningTime="2026-07-12 17:10:39.368880277 +0000 UTC m=+447.489197075"
Jul 12 17:10:50.343995 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:10:50.343958    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-hl4zn"
Jul 12 17:11:07.445931 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:11:07.445896    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-kfk5j"]
Jul 12 17:11:07.446335 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:11:07.446063    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-kfk5j" podUID="29eca76d-1ab5-43a5-a4da-af12c349cae0" containerName="authorino" containerID="cri-o://09515853e02f18ea15d1179c93749a87fb7670c6fe3029890a3b773316644d20" gracePeriod=30
Jul 12 17:11:07.729647 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:11:07.729586    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-kfk5j"
Jul 12 17:11:07.775292 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:11:07.775262    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8vhrv\" (UniqueName: \"kubernetes.io/projected/29eca76d-1ab5-43a5-a4da-af12c349cae0-kube-api-access-8vhrv\") pod \"29eca76d-1ab5-43a5-a4da-af12c349cae0\" (UID: \"29eca76d-1ab5-43a5-a4da-af12c349cae0\") "
Jul 12 17:11:07.777283 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:11:07.777257    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29eca76d-1ab5-43a5-a4da-af12c349cae0-kube-api-access-8vhrv" (OuterVolumeSpecName: "kube-api-access-8vhrv") pod "29eca76d-1ab5-43a5-a4da-af12c349cae0" (UID: "29eca76d-1ab5-43a5-a4da-af12c349cae0"). InnerVolumeSpecName "kube-api-access-8vhrv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 12 17:11:07.876337 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:11:07.876308    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vhrv\" (UniqueName: \"kubernetes.io/projected/29eca76d-1ab5-43a5-a4da-af12c349cae0-kube-api-access-8vhrv\") on node \"ip-10-0-141-94.ec2.internal\" DevicePath \"\""
Jul 12 17:11:08.452285 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:11:08.452255    2569 generic.go:356] "Generic (PLEG): container finished" podID="29eca76d-1ab5-43a5-a4da-af12c349cae0" containerID="09515853e02f18ea15d1179c93749a87fb7670c6fe3029890a3b773316644d20" exitCode=0
Jul 12 17:11:08.452670 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:11:08.452292    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-kfk5j"
Jul 12 17:11:08.452670 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:11:08.452294    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-kfk5j" event={"ID":"29eca76d-1ab5-43a5-a4da-af12c349cae0","Type":"ContainerDied","Data":"09515853e02f18ea15d1179c93749a87fb7670c6fe3029890a3b773316644d20"}
Jul 12 17:11:08.452670 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:11:08.452426    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-kfk5j" event={"ID":"29eca76d-1ab5-43a5-a4da-af12c349cae0","Type":"ContainerDied","Data":"89d1bd134c0a66d7bea21a9928bd3112e99bdabed2f477e87674a1c290ae4243"}
Jul 12 17:11:08.452670 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:11:08.452452    2569 scope.go:117] "RemoveContainer" containerID="09515853e02f18ea15d1179c93749a87fb7670c6fe3029890a3b773316644d20"
Jul 12 17:11:08.462251 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:11:08.462229    2569 scope.go:117] "RemoveContainer" containerID="09515853e02f18ea15d1179c93749a87fb7670c6fe3029890a3b773316644d20"
Jul 12 17:11:08.462604 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:11:08.462582    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09515853e02f18ea15d1179c93749a87fb7670c6fe3029890a3b773316644d20\": container with ID starting with 09515853e02f18ea15d1179c93749a87fb7670c6fe3029890a3b773316644d20 not found: ID does not exist" containerID="09515853e02f18ea15d1179c93749a87fb7670c6fe3029890a3b773316644d20"
Jul 12 17:11:08.462675 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:11:08.462614    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09515853e02f18ea15d1179c93749a87fb7670c6fe3029890a3b773316644d20"} err="failed to get container status \"09515853e02f18ea15d1179c93749a87fb7670c6fe3029890a3b773316644d20\": rpc error: code = NotFound desc = could not find container \"09515853e02f18ea15d1179c93749a87fb7670c6fe3029890a3b773316644d20\": container with ID starting with 09515853e02f18ea15d1179c93749a87fb7670c6fe3029890a3b773316644d20 not found: ID does not exist"
Jul 12 17:11:08.468933 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:11:08.468913    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-kfk5j"]
Jul 12 17:11:08.475030 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:11:08.475011    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-kfk5j"]
Jul 12 17:11:10.379160 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:11:10.379130    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="29eca76d-1ab5-43a5-a4da-af12c349cae0" path="/var/lib/kubelet/pods/29eca76d-1ab5-43a5-a4da-af12c349cae0/volumes"
Jul 12 17:11:12.273113 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:11:12.273086    2569 scope.go:117] "RemoveContainer" containerID="3ccb29edf5f2c04eb590b42afe6949d4a28420b03011a2b20a5a295c5a0e98c0"
Jul 12 17:11:48.195780 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:11:48.195742    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl4zn"]
Jul 12 17:12:07.782032 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:12:07.781957    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4ldrp"]
Jul 12 17:12:07.782454 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:12:07.782260    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29eca76d-1ab5-43a5-a4da-af12c349cae0" containerName="authorino"
Jul 12 17:12:07.782454 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:12:07.782270    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="29eca76d-1ab5-43a5-a4da-af12c349cae0" containerName="authorino"
Jul 12 17:12:07.782454 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:12:07.782316    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="29eca76d-1ab5-43a5-a4da-af12c349cae0" containerName="authorino"
Jul 12 17:12:07.784044 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:12:07.784028    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4ldrp"
Jul 12 17:12:07.786386 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:12:07.786361    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 12 17:12:07.787306 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:12:07.787287    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-5kh7x\""
Jul 12 17:12:07.787306 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:12:07.787299    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Jul 12 17:12:07.787461 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:12:07.787288    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 12 17:12:07.794439 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:12:07.794415    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4ldrp"]
Jul 12 17:12:07.825819 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:12:07.825793    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/994b775c-7cf7-449e-9f40-17e7b9afa9b5-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4ldrp\" (UID: \"994b775c-7cf7-449e-9f40-17e7b9afa9b5\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4ldrp"
Jul 12 17:12:07.825917 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:12:07.825830    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/994b775c-7cf7-449e-9f40-17e7b9afa9b5-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4ldrp\" (UID: \"994b775c-7cf7-449e-9f40-17e7b9afa9b5\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4ldrp"
Jul 12 17:12:07.825917 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:12:07.825850    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/994b775c-7cf7-449e-9f40-17e7b9afa9b5-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4ldrp\" (UID: \"994b775c-7cf7-449e-9f40-17e7b9afa9b5\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4ldrp"
Jul 12 17:12:07.825917 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:12:07.825901    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/994b775c-7cf7-449e-9f40-17e7b9afa9b5-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4ldrp\" (UID: \"994b775c-7cf7-449e-9f40-17e7b9afa9b5\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4ldrp"
Jul 12 17:12:07.826023 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:12:07.825922    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tz7xn\" (UniqueName: \"kubernetes.io/projected/994b775c-7cf7-449e-9f40-17e7b9afa9b5-kube-api-access-tz7xn\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4ldrp\" (UID: \"994b775c-7cf7-449e-9f40-17e7b9afa9b5\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4ldrp"
Jul 12 17:12:07.826023 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:12:07.825940    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/994b775c-7cf7-449e-9f40-17e7b9afa9b5-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4ldrp\" (UID: \"994b775c-7cf7-449e-9f40-17e7b9afa9b5\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4ldrp"
Jul 12 17:12:07.927064 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:12:07.927037    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/994b775c-7cf7-449e-9f40-17e7b9afa9b5-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4ldrp\" (UID: \"994b775c-7cf7-449e-9f40-17e7b9afa9b5\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4ldrp"
Jul 12 17:12:07.927158 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:12:07.927073    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/994b775c-7cf7-449e-9f40-17e7b9afa9b5-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4ldrp\" (UID: \"994b775c-7cf7-449e-9f40-17e7b9afa9b5\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4ldrp"
Jul 12 17:12:07.927158 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:12:07.927090    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/994b775c-7cf7-449e-9f40-17e7b9afa9b5-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4ldrp\" (UID: \"994b775c-7cf7-449e-9f40-17e7b9afa9b5\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4ldrp"
Jul 12 17:12:07.927158 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:12:07.927113    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/994b775c-7cf7-449e-9f40-17e7b9afa9b5-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4ldrp\" (UID: \"994b775c-7cf7-449e-9f40-17e7b9afa9b5\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4ldrp"
Jul 12 17:12:07.927158 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:12:07.927131    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tz7xn\" (UniqueName: \"kubernetes.io/projected/994b775c-7cf7-449e-9f40-17e7b9afa9b5-kube-api-access-tz7xn\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4ldrp\" (UID: \"994b775c-7cf7-449e-9f40-17e7b9afa9b5\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4ldrp"
Jul 12 17:12:07.927158 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:12:07.927150    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/994b775c-7cf7-449e-9f40-17e7b9afa9b5-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4ldrp\" (UID: \"994b775c-7cf7-449e-9f40-17e7b9afa9b5\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4ldrp"
Jul 12 17:12:07.927480 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:12:07.927461    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/994b775c-7cf7-449e-9f40-17e7b9afa9b5-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4ldrp\" (UID: \"994b775c-7cf7-449e-9f40-17e7b9afa9b5\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4ldrp"
Jul 12 17:12:07.927577 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:12:07.927477    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/994b775c-7cf7-449e-9f40-17e7b9afa9b5-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4ldrp\" (UID: \"994b775c-7cf7-449e-9f40-17e7b9afa9b5\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4ldrp"
Jul 12 17:12:07.927577 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:12:07.927542    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/994b775c-7cf7-449e-9f40-17e7b9afa9b5-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4ldrp\" (UID: \"994b775c-7cf7-449e-9f40-17e7b9afa9b5\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4ldrp"
Jul 12 17:12:07.929374 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:12:07.929349    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/994b775c-7cf7-449e-9f40-17e7b9afa9b5-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4ldrp\" (UID: \"994b775c-7cf7-449e-9f40-17e7b9afa9b5\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4ldrp"
Jul 12 17:12:07.929527 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:12:07.929490    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/994b775c-7cf7-449e-9f40-17e7b9afa9b5-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4ldrp\" (UID: \"994b775c-7cf7-449e-9f40-17e7b9afa9b5\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4ldrp"
Jul 12 17:12:07.935635 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:12:07.935614    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tz7xn\" (UniqueName: \"kubernetes.io/projected/994b775c-7cf7-449e-9f40-17e7b9afa9b5-kube-api-access-tz7xn\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4ldrp\" (UID: \"994b775c-7cf7-449e-9f40-17e7b9afa9b5\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4ldrp"
Jul 12 17:12:08.095860 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:12:08.095779    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4ldrp"
Jul 12 17:12:08.219974 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:12:08.219950    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4ldrp"]
Jul 12 17:12:08.221893 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:12:08.221867    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod994b775c_7cf7_449e_9f40_17e7b9afa9b5.slice/crio-f6d44de545d001f2818e3cfb17a5e1039161b0e775e8cc1b151759ced63d2d2c WatchSource:0}: Error finding container f6d44de545d001f2818e3cfb17a5e1039161b0e775e8cc1b151759ced63d2d2c: Status 404 returned error can't find the container with id f6d44de545d001f2818e3cfb17a5e1039161b0e775e8cc1b151759ced63d2d2c
Jul 12 17:12:08.289610 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:12:08.289584    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl4zn"]
Jul 12 17:12:08.693987 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:12:08.693944    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4ldrp" event={"ID":"994b775c-7cf7-449e-9f40-17e7b9afa9b5","Type":"ContainerStarted","Data":"f6d44de545d001f2818e3cfb17a5e1039161b0e775e8cc1b151759ced63d2d2c"}
Jul 12 17:12:09.699933 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:12:09.699898    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4ldrp" event={"ID":"994b775c-7cf7-449e-9f40-17e7b9afa9b5","Type":"ContainerStarted","Data":"4ea5b64723ba6417c7595abc1cdf38df777b8dbef3b78ada9645ff7285bd0e1b"}
Jul 12 17:12:09.717973 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:12:09.717897    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4ldrp" podStartSLOduration=1.56352344 podStartE2EDuration="2.717883219s" totalImagesPullingTime="1.154359779s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 17:12:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 17:12:09.716727948 +0000 UTC m=+537.837044745" watchObservedRunningTime="2026-07-12 17:12:09.717883219 +0000 UTC m=+537.838200015"
Jul 12 17:12:16.184144 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:12:16.184100    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl4zn"]
Jul 12 17:12:18.096813 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:12:18.096782    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4ldrp"
Jul 12 17:12:18.097256 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:12:18.096861    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4ldrp"
Jul 12 17:12:18.109099 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:12:18.109076    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4ldrp"
Jul 12 17:12:18.747051 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:12:18.747025    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4ldrp"
Jul 12 17:12:24.288032 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:12:24.287998    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl4zn"]
Jul 12 17:12:34.501825 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:12:34.501794    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl4zn"]
Jul 12 17:12:35.787178 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:12:35.787147    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-nvspc"]
Jul 12 17:12:35.790433 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:12:35.790411    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-nvspc"
Jul 12 17:12:35.793034 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:12:35.793014    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Jul 12 17:12:35.802130 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:12:35.802108    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-nvspc"]
Jul 12 17:12:35.937569 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:12:35.937542    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/981e1044-8c89-4da6-8476-6828123a6efc-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-nvspc\" (UID: \"981e1044-8c89-4da6-8476-6828123a6efc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-nvspc"
Jul 12 17:12:35.937695 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:12:35.937590    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6749s\" (UniqueName: \"kubernetes.io/projected/981e1044-8c89-4da6-8476-6828123a6efc-kube-api-access-6749s\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-nvspc\" (UID: \"981e1044-8c89-4da6-8476-6828123a6efc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-nvspc"
Jul 12 17:12:35.937695 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:12:35.937614    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/981e1044-8c89-4da6-8476-6828123a6efc-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-nvspc\" (UID: \"981e1044-8c89-4da6-8476-6828123a6efc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-nvspc"
Jul 12 17:12:35.937816 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:12:35.937689    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/981e1044-8c89-4da6-8476-6828123a6efc-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-nvspc\" (UID: \"981e1044-8c89-4da6-8476-6828123a6efc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-nvspc"
Jul 12 17:12:35.937816 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:12:35.937738    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/981e1044-8c89-4da6-8476-6828123a6efc-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-nvspc\" (UID: \"981e1044-8c89-4da6-8476-6828123a6efc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-nvspc"
Jul 12 17:12:35.937816 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:12:35.937792    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/981e1044-8c89-4da6-8476-6828123a6efc-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-nvspc\" (UID: \"981e1044-8c89-4da6-8476-6828123a6efc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-nvspc"
Jul 12 17:12:36.038531 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:12:36.038429    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6749s\" (UniqueName: \"kubernetes.io/projected/981e1044-8c89-4da6-8476-6828123a6efc-kube-api-access-6749s\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-nvspc\" (UID: \"981e1044-8c89-4da6-8476-6828123a6efc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-nvspc"
Jul 12 17:12:36.038531 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:12:36.038469    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/981e1044-8c89-4da6-8476-6828123a6efc-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-nvspc\" (UID: \"981e1044-8c89-4da6-8476-6828123a6efc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-nvspc"
Jul 12 17:12:36.038531 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:12:36.038497    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/981e1044-8c89-4da6-8476-6828123a6efc-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-nvspc\" (UID: \"981e1044-8c89-4da6-8476-6828123a6efc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-nvspc"
Jul 12 17:12:36.038763 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:12:36.038563    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/981e1044-8c89-4da6-8476-6828123a6efc-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-nvspc\" (UID: \"981e1044-8c89-4da6-8476-6828123a6efc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-nvspc"
Jul 12 17:12:36.038763 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:12:36.038606    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/981e1044-8c89-4da6-8476-6828123a6efc-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-nvspc\" (UID: \"981e1044-8c89-4da6-8476-6828123a6efc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-nvspc"
Jul 12 17:12:36.038763 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:12:36.038641    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/981e1044-8c89-4da6-8476-6828123a6efc-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-nvspc\" (UID: \"981e1044-8c89-4da6-8476-6828123a6efc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-nvspc"
Jul 12 17:12:36.038916 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:12:36.038886    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/981e1044-8c89-4da6-8476-6828123a6efc-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-nvspc\" (UID: \"981e1044-8c89-4da6-8476-6828123a6efc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-nvspc"
Jul 12 17:12:36.038973 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:12:36.038940    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/981e1044-8c89-4da6-8476-6828123a6efc-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-nvspc\" (UID: \"981e1044-8c89-4da6-8476-6828123a6efc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-nvspc"
Jul 12 17:12:36.039029 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:12:36.039011    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/981e1044-8c89-4da6-8476-6828123a6efc-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-nvspc\" (UID: \"981e1044-8c89-4da6-8476-6828123a6efc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-nvspc"
Jul 12 17:12:36.040863 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:12:36.040841    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/981e1044-8c89-4da6-8476-6828123a6efc-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-nvspc\" (UID: \"981e1044-8c89-4da6-8476-6828123a6efc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-nvspc"
Jul 12 17:12:36.040967 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:12:36.040957    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/981e1044-8c89-4da6-8476-6828123a6efc-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-nvspc\" (UID: \"981e1044-8c89-4da6-8476-6828123a6efc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-nvspc"
Jul 12 17:12:36.051109 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:12:36.051082    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6749s\" (UniqueName: \"kubernetes.io/projected/981e1044-8c89-4da6-8476-6828123a6efc-kube-api-access-6749s\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-nvspc\" (UID: \"981e1044-8c89-4da6-8476-6828123a6efc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-nvspc"
Jul 12 17:12:36.099927 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:12:36.099899    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-nvspc"
Jul 12 17:12:36.225317 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:12:36.225260    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-nvspc"]
Jul 12 17:12:36.809554 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:12:36.809525    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-nvspc" event={"ID":"981e1044-8c89-4da6-8476-6828123a6efc","Type":"ContainerStarted","Data":"0e87c69bdc8cc13012fa1fc122d387fde83709af00af1f2de2d8fdbfbfac227b"}
Jul 12 17:12:36.809916 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:12:36.809577    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-nvspc" event={"ID":"981e1044-8c89-4da6-8476-6828123a6efc","Type":"ContainerStarted","Data":"2faf240a3b62bb4e74aca25d12fcf5872fc42013386e77f7397114c8f2077016"}
Jul 12 17:12:36.826898 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:12:36.826843    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-nvspc" podStartSLOduration=1.64808116 podStartE2EDuration="1.826828372s" totalImagesPullingTime="178.747212ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 17:12:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 17:12:36.825491028 +0000 UTC m=+564.945807837" watchObservedRunningTime="2026-07-12 17:12:36.826828372 +0000 UTC m=+564.947145169"
Jul 12 17:12:41.375290 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:12:41.375251    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tc285"]
Jul 12 17:12:41.379870 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:12:41.379851    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tc285"
Jul 12 17:12:41.382054 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:12:41.382033    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Jul 12 17:12:41.388992 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:12:41.388964    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tc285"]
Jul 12 17:12:41.484367 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:12:41.484342    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5cf5611-64a5-4ace-a2c2-16d2d52b32da-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-tc285\" (UID: \"f5cf5611-64a5-4ace-a2c2-16d2d52b32da\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tc285"
Jul 12 17:12:41.484528 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:12:41.484393    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vth45\" (UniqueName: \"kubernetes.io/projected/f5cf5611-64a5-4ace-a2c2-16d2d52b32da-kube-api-access-vth45\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-tc285\" (UID: \"f5cf5611-64a5-4ace-a2c2-16d2d52b32da\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tc285"
Jul 12 17:12:41.484528 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:12:41.484452    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5cf5611-64a5-4ace-a2c2-16d2d52b32da-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-tc285\" (UID: \"f5cf5611-64a5-4ace-a2c2-16d2d52b32da\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tc285"
Jul 12 17:12:41.484528 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:12:41.484474    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5cf5611-64a5-4ace-a2c2-16d2d52b32da-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-tc285\" (UID: \"f5cf5611-64a5-4ace-a2c2-16d2d52b32da\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tc285"
Jul 12 17:12:41.484528 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:12:41.484498    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5cf5611-64a5-4ace-a2c2-16d2d52b32da-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-tc285\" (UID: \"f5cf5611-64a5-4ace-a2c2-16d2d52b32da\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tc285"
Jul 12 17:12:41.484692 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:12:41.484586    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5cf5611-64a5-4ace-a2c2-16d2d52b32da-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-tc285\" (UID: \"f5cf5611-64a5-4ace-a2c2-16d2d52b32da\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tc285"
Jul 12 17:12:41.585091 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:12:41.585065    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5cf5611-64a5-4ace-a2c2-16d2d52b32da-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-tc285\" (UID: \"f5cf5611-64a5-4ace-a2c2-16d2d52b32da\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tc285"
Jul 12 17:12:41.585192 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:12:41.585100    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vth45\" (UniqueName: \"kubernetes.io/projected/f5cf5611-64a5-4ace-a2c2-16d2d52b32da-kube-api-access-vth45\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-tc285\" (UID: \"f5cf5611-64a5-4ace-a2c2-16d2d52b32da\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tc285"
Jul 12 17:12:41.585192 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:12:41.585133    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5cf5611-64a5-4ace-a2c2-16d2d52b32da-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-tc285\" (UID: \"f5cf5611-64a5-4ace-a2c2-16d2d52b32da\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tc285"
Jul 12 17:12:41.585192 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:12:41.585153    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5cf5611-64a5-4ace-a2c2-16d2d52b32da-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-tc285\" (UID: \"f5cf5611-64a5-4ace-a2c2-16d2d52b32da\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tc285"
Jul 12 17:12:41.585192 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:12:41.585176    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5cf5611-64a5-4ace-a2c2-16d2d52b32da-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-tc285\" (UID: \"f5cf5611-64a5-4ace-a2c2-16d2d52b32da\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tc285"
Jul 12 17:12:41.585192 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:12:41.585192    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5cf5611-64a5-4ace-a2c2-16d2d52b32da-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-tc285\" (UID: \"f5cf5611-64a5-4ace-a2c2-16d2d52b32da\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tc285"
Jul 12 17:12:41.585614 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:12:41.585595    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5cf5611-64a5-4ace-a2c2-16d2d52b32da-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-tc285\" (UID: \"f5cf5611-64a5-4ace-a2c2-16d2d52b32da\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tc285"
Jul 12 17:12:41.585681 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:12:41.585637    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5cf5611-64a5-4ace-a2c2-16d2d52b32da-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-tc285\" (UID: \"f5cf5611-64a5-4ace-a2c2-16d2d52b32da\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tc285"
Jul 12 17:12:41.585681 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:12:41.585659    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5cf5611-64a5-4ace-a2c2-16d2d52b32da-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-tc285\" (UID: \"f5cf5611-64a5-4ace-a2c2-16d2d52b32da\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tc285"
Jul 12 17:12:41.587467 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:12:41.587449    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5cf5611-64a5-4ace-a2c2-16d2d52b32da-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-tc285\" (UID: \"f5cf5611-64a5-4ace-a2c2-16d2d52b32da\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tc285"
Jul 12 17:12:41.587810 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:12:41.587794    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5cf5611-64a5-4ace-a2c2-16d2d52b32da-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-tc285\" (UID: \"f5cf5611-64a5-4ace-a2c2-16d2d52b32da\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tc285"
Jul 12 17:12:41.593134 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:12:41.593113    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vth45\" (UniqueName: \"kubernetes.io/projected/f5cf5611-64a5-4ace-a2c2-16d2d52b32da-kube-api-access-vth45\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-tc285\" (UID: \"f5cf5611-64a5-4ace-a2c2-16d2d52b32da\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tc285"
Jul 12 17:12:41.689254 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:12:41.689217    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tc285"
Jul 12 17:12:41.812401 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:12:41.812249    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tc285"]
Jul 12 17:12:41.815150 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:12:41.815109    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf5cf5611_64a5_4ace_a2c2_16d2d52b32da.slice/crio-efd65d0c2e092510a5eadbe7d6c486f623d1e1d564226ba40a97f7532e117df9 WatchSource:0}: Error finding container efd65d0c2e092510a5eadbe7d6c486f623d1e1d564226ba40a97f7532e117df9: Status 404 returned error can't find the container with id efd65d0c2e092510a5eadbe7d6c486f623d1e1d564226ba40a97f7532e117df9
Jul 12 17:12:41.832182 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:12:41.832151    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tc285" event={"ID":"f5cf5611-64a5-4ace-a2c2-16d2d52b32da","Type":"ContainerStarted","Data":"efd65d0c2e092510a5eadbe7d6c486f623d1e1d564226ba40a97f7532e117df9"}
Jul 12 17:12:42.837207 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:12:42.837169    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tc285" event={"ID":"f5cf5611-64a5-4ace-a2c2-16d2d52b32da","Type":"ContainerStarted","Data":"f8df2cba99a6f6ae6113c5531adbc43e644f3139fdf877db9c2fefe384ca4d1c"}
Jul 12 17:12:42.857268 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:12:42.857224    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tc285" podStartSLOduration=1.6766926199999999 podStartE2EDuration="1.857209676s" totalImagesPullingTime="180.517056ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 17:12:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 17:12:42.855066079 +0000 UTC m=+570.975382889" watchObservedRunningTime="2026-07-12 17:12:42.857209676 +0000 UTC m=+570.977526473"
Jul 12 17:12:43.192606 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:12:43.192573    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl4zn"]
Jul 12 17:12:46.101036 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:12:46.100994    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-nvspc"
Jul 12 17:12:46.101481 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:12:46.101104    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-nvspc"
Jul 12 17:12:46.113555 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:12:46.113531    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-nvspc"
Jul 12 17:12:46.865326 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:12:46.865293    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-nvspc"
Jul 12 17:12:51.689615 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:12:51.689577    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tc285"
Jul 12 17:12:51.689615 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:12:51.689622    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tc285"
Jul 12 17:12:51.702114 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:12:51.702091    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tc285"
Jul 12 17:12:51.886361 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:12:51.886335    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tc285"
Jul 12 17:13:12.275946 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:13:12.275914    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9xbrj_6b618491-4e81-4708-ab1b-2781d1df7c87/ovn-acl-logging/0.log"
Jul 12 17:13:12.284676 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:13:12.284653    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9xbrj_6b618491-4e81-4708-ab1b-2781d1df7c87/ovn-acl-logging/0.log"
Jul 12 17:14:03.896199 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:14:03.896160    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl4zn"]
Jul 12 17:14:13.593790 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:14:13.593760    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl4zn"]
Jul 12 17:14:23.189995 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:14:23.189964    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl4zn"]
Jul 12 17:14:31.790571 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:14:31.790537    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl4zn"]
Jul 12 17:14:43.084242 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:14:43.084207    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl4zn"]
Jul 12 17:15:13.000819 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:15:13.000737    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl4zn"]
Jul 12 17:15:25.394995 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:15:25.394960    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl4zn"]
Jul 12 17:16:00.894040 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:16:00.894001    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl4zn"]
Jul 12 17:16:09.183893 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:16:09.183855    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl4zn"]
Jul 12 17:16:19.193307 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:16:19.193272    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl4zn"]
Jul 12 17:16:24.788140 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:16:24.788105    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl4zn"]
Jul 12 17:16:35.986838 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:16:35.986753    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl4zn"]
Jul 12 17:16:46.213569 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:16:46.213534    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl4zn"]
Jul 12 17:16:56.988614 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:16:56.988583    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl4zn"]
Jul 12 17:17:06.888678 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:17:06.888643    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl4zn"]
Jul 12 17:17:28.788050 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:17:28.788018    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl4zn"]
Jul 12 17:17:40.089877 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:17:40.089847    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl4zn"]
Jul 12 17:18:04.192642 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:18:04.192569    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl4zn"]
Jul 12 17:18:12.301470 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:18:12.301443    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9xbrj_6b618491-4e81-4708-ab1b-2781d1df7c87/ovn-acl-logging/0.log"
Jul 12 17:18:12.307250 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:18:12.307231    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9xbrj_6b618491-4e81-4708-ab1b-2781d1df7c87/ovn-acl-logging/0.log"
Jul 12 17:18:15.186604 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:18:15.186571    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl4zn"]
Jul 12 17:18:29.992821 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:18:29.992792    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl4zn"]
Jul 12 17:18:39.489912 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:18:39.489877    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl4zn"]
Jul 12 17:18:50.795933 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:18:50.795898    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl4zn"]
Jul 12 17:19:03.185388 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:19:03.185351    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl4zn"]
Jul 12 17:20:56.185560 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:20:56.185525    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl4zn"]
Jul 12 17:21:05.781042 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:21:05.780963    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl4zn"]
Jul 12 17:21:19.089218 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:21:19.089175    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl4zn"]
Jul 12 17:21:28.295663 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:21:28.295628    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl4zn"]
Jul 12 17:21:45.991592 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:21:45.991556    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl4zn"]
Jul 12 17:21:56.291839 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:21:56.291803    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl4zn"]
Jul 12 17:22:07.879807 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:22:07.879776    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl4zn"]
Jul 12 17:22:16.586118 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:22:16.586081    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl4zn"]
Jul 12 17:22:26.589383 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:22:26.589348    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl4zn"]
Jul 12 17:23:04.886742 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:23:04.886665    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl4zn"]
Jul 12 17:23:12.332336 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:23:12.332305    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9xbrj_6b618491-4e81-4708-ab1b-2781d1df7c87/ovn-acl-logging/0.log"
Jul 12 17:23:12.334182 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:23:12.334161    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9xbrj_6b618491-4e81-4708-ab1b-2781d1df7c87/ovn-acl-logging/0.log"
Jul 12 17:23:14.397047 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:23:14.397010    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl4zn"]
Jul 12 17:23:24.091643 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:23:24.091609    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl4zn"]
Jul 12 17:23:36.082139 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:23:36.082101    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl4zn"]
Jul 12 17:23:52.484480 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:23:52.484442    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl4zn"]
Jul 12 17:24:01.988110 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:24:01.988078    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl4zn"]
Jul 12 17:24:11.783946 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:24:11.783870    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl4zn"]
Jul 12 17:24:19.895953 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:24:19.895918    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl4zn"]
Jul 12 17:24:30.386933 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:24:30.386902    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl4zn"]
Jul 12 17:24:40.189289 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:24:40.189256    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl4zn"]
Jul 12 17:24:50.782735 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:24:50.782703    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl4zn"]
Jul 12 17:25:02.398407 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:25:02.398375    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl4zn"]
Jul 12 17:25:12.481569 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:25:12.481535    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl4zn"]
Jul 12 17:25:25.093660 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:25:25.093628    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl4zn"]
Jul 12 17:25:34.491670 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:25:34.491595    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl4zn"]
Jul 12 17:25:41.886598 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:25:41.886563    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl4zn"]
Jul 12 17:25:58.494321 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:25:58.494284    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl4zn"]
Jul 12 17:26:10.386076 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:26:10.386042    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl4zn"]
Jul 12 17:26:24.386014 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:26:24.385982    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl4zn"]
Jul 12 17:26:37.494451 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:26:37.494413    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl4zn"]
Jul 12 17:26:49.285998 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:26:49.285964    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl4zn"]
Jul 12 17:26:59.689797 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:26:59.689765    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl4zn"]
Jul 12 17:27:14.988452 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:27:14.988373    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl4zn"]
Jul 12 17:27:31.288234 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:27:31.288196    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl4zn"]
Jul 12 17:28:12.363361 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:28:12.363330    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9xbrj_6b618491-4e81-4708-ab1b-2781d1df7c87/ovn-acl-logging/0.log"
Jul 12 17:28:12.363361 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:28:12.363347    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9xbrj_6b618491-4e81-4708-ab1b-2781d1df7c87/ovn-acl-logging/0.log"
Jul 12 17:28:36.585975 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:28:36.585904    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl4zn"]
Jul 12 17:28:44.779819 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:28:44.779786    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl4zn"]
Jul 12 17:28:59.979685 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:28:59.979648    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl4zn"]
Jul 12 17:29:17.184463 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:29:17.184431    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl4zn"]
Jul 12 17:30:31.888599 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:30:31.888567    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl4zn"]
Jul 12 17:32:06.390714 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:32:06.390641    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl4zn"]
Jul 12 17:32:15.780455 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:32:15.780417    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl4zn"]
Jul 12 17:32:28.492995 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:32:28.492961    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl4zn"]
Jul 12 17:32:48.093459 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:32:48.093422    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl4zn"]
Jul 12 17:33:12.388549 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:33:12.388502    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9xbrj_6b618491-4e81-4708-ab1b-2781d1df7c87/ovn-acl-logging/0.log"
Jul 12 17:33:12.390428 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:33:12.390409    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9xbrj_6b618491-4e81-4708-ab1b-2781d1df7c87/ovn-acl-logging/0.log"
Jul 12 17:33:29.579326 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:33:29.579291    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl4zn"]
Jul 12 17:33:38.595126 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:33:38.595094    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-d91d15cb/e2e-lifecycle-model-d91d15cb-kserve-7d4b9fb6f9-85t64"]
Jul 12 17:33:38.598706 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:33:38.598687    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-d91d15cb/e2e-lifecycle-model-d91d15cb-kserve-7d4b9fb6f9-85t64"
Jul 12 17:33:38.601755 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:33:38.601729    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-d91d15cb\"/\"e2e-lifecycle-model-d91d15cb-kserve-self-signed-certs\""
Jul 12 17:33:38.601755 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:33:38.601748    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-d91d15cb\"/\"default-dockercfg-fbx5k\""
Jul 12 17:33:38.601928 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:33:38.601735    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-d91d15cb\"/\"openshift-service-ca.crt\""
Jul 12 17:33:38.601928 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:33:38.601784    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-d91d15cb\"/\"kube-root-ca.crt\""
Jul 12 17:33:38.606533 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:33:38.606493    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-d91d15cb/e2e-lifecycle-model-d91d15cb-kserve-7d4b9fb6f9-85t64"]
Jul 12 17:33:38.690677 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:33:38.690644    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8852\" (UniqueName: \"kubernetes.io/projected/179468b3-b372-4905-87d4-1de35ed0ce2e-kube-api-access-p8852\") pod \"e2e-lifecycle-model-d91d15cb-kserve-7d4b9fb6f9-85t64\" (UID: \"179468b3-b372-4905-87d4-1de35ed0ce2e\") " pod="ai-tenant-e2e-mt-d91d15cb/e2e-lifecycle-model-d91d15cb-kserve-7d4b9fb6f9-85t64"
Jul 12 17:33:38.690677 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:33:38.690681    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/179468b3-b372-4905-87d4-1de35ed0ce2e-tmp-dir\") pod \"e2e-lifecycle-model-d91d15cb-kserve-7d4b9fb6f9-85t64\" (UID: \"179468b3-b372-4905-87d4-1de35ed0ce2e\") " pod="ai-tenant-e2e-mt-d91d15cb/e2e-lifecycle-model-d91d15cb-kserve-7d4b9fb6f9-85t64"
Jul 12 17:33:38.690876 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:33:38.690708    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/179468b3-b372-4905-87d4-1de35ed0ce2e-model-cache\") pod \"e2e-lifecycle-model-d91d15cb-kserve-7d4b9fb6f9-85t64\" (UID: \"179468b3-b372-4905-87d4-1de35ed0ce2e\") " pod="ai-tenant-e2e-mt-d91d15cb/e2e-lifecycle-model-d91d15cb-kserve-7d4b9fb6f9-85t64"
Jul 12 17:33:38.690876 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:33:38.690725    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/179468b3-b372-4905-87d4-1de35ed0ce2e-tls-certs\") pod \"e2e-lifecycle-model-d91d15cb-kserve-7d4b9fb6f9-85t64\" (UID: \"179468b3-b372-4905-87d4-1de35ed0ce2e\") " pod="ai-tenant-e2e-mt-d91d15cb/e2e-lifecycle-model-d91d15cb-kserve-7d4b9fb6f9-85t64"
Jul 12 17:33:38.690876 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:33:38.690831    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/179468b3-b372-4905-87d4-1de35ed0ce2e-dshm\") pod \"e2e-lifecycle-model-d91d15cb-kserve-7d4b9fb6f9-85t64\" (UID: \"179468b3-b372-4905-87d4-1de35ed0ce2e\") " pod="ai-tenant-e2e-mt-d91d15cb/e2e-lifecycle-model-d91d15cb-kserve-7d4b9fb6f9-85t64"
Jul 12 17:33:38.690876 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:33:38.690866    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/179468b3-b372-4905-87d4-1de35ed0ce2e-home\") pod \"e2e-lifecycle-model-d91d15cb-kserve-7d4b9fb6f9-85t64\" (UID: \"179468b3-b372-4905-87d4-1de35ed0ce2e\") " pod="ai-tenant-e2e-mt-d91d15cb/e2e-lifecycle-model-d91d15cb-kserve-7d4b9fb6f9-85t64"
Jul 12 17:33:38.792345 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:33:38.792311    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/179468b3-b372-4905-87d4-1de35ed0ce2e-dshm\") pod \"e2e-lifecycle-model-d91d15cb-kserve-7d4b9fb6f9-85t64\" (UID: \"179468b3-b372-4905-87d4-1de35ed0ce2e\") " pod="ai-tenant-e2e-mt-d91d15cb/e2e-lifecycle-model-d91d15cb-kserve-7d4b9fb6f9-85t64"
Jul 12 17:33:38.792345 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:33:38.792352    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/179468b3-b372-4905-87d4-1de35ed0ce2e-home\") pod \"e2e-lifecycle-model-d91d15cb-kserve-7d4b9fb6f9-85t64\" (UID: \"179468b3-b372-4905-87d4-1de35ed0ce2e\") " pod="ai-tenant-e2e-mt-d91d15cb/e2e-lifecycle-model-d91d15cb-kserve-7d4b9fb6f9-85t64"
Jul 12 17:33:38.792591 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:33:38.792455    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8852\" (UniqueName: \"kubernetes.io/projected/179468b3-b372-4905-87d4-1de35ed0ce2e-kube-api-access-p8852\") pod \"e2e-lifecycle-model-d91d15cb-kserve-7d4b9fb6f9-85t64\" (UID: \"179468b3-b372-4905-87d4-1de35ed0ce2e\") " pod="ai-tenant-e2e-mt-d91d15cb/e2e-lifecycle-model-d91d15cb-kserve-7d4b9fb6f9-85t64"
Jul 12 17:33:38.792591 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:33:38.792476    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/179468b3-b372-4905-87d4-1de35ed0ce2e-tmp-dir\") pod \"e2e-lifecycle-model-d91d15cb-kserve-7d4b9fb6f9-85t64\" (UID: \"179468b3-b372-4905-87d4-1de35ed0ce2e\") " pod="ai-tenant-e2e-mt-d91d15cb/e2e-lifecycle-model-d91d15cb-kserve-7d4b9fb6f9-85t64"
Jul 12 17:33:38.792591 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:33:38.792500    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/179468b3-b372-4905-87d4-1de35ed0ce2e-model-cache\") pod \"e2e-lifecycle-model-d91d15cb-kserve-7d4b9fb6f9-85t64\" (UID: \"179468b3-b372-4905-87d4-1de35ed0ce2e\") " pod="ai-tenant-e2e-mt-d91d15cb/e2e-lifecycle-model-d91d15cb-kserve-7d4b9fb6f9-85t64"
Jul 12 17:33:38.792591 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:33:38.792543    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/179468b3-b372-4905-87d4-1de35ed0ce2e-tls-certs\") pod \"e2e-lifecycle-model-d91d15cb-kserve-7d4b9fb6f9-85t64\" (UID: \"179468b3-b372-4905-87d4-1de35ed0ce2e\") " pod="ai-tenant-e2e-mt-d91d15cb/e2e-lifecycle-model-d91d15cb-kserve-7d4b9fb6f9-85t64"
Jul 12 17:33:38.792864 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:33:38.792840    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/179468b3-b372-4905-87d4-1de35ed0ce2e-home\") pod \"e2e-lifecycle-model-d91d15cb-kserve-7d4b9fb6f9-85t64\" (UID: \"179468b3-b372-4905-87d4-1de35ed0ce2e\") " pod="ai-tenant-e2e-mt-d91d15cb/e2e-lifecycle-model-d91d15cb-kserve-7d4b9fb6f9-85t64"
Jul 12 17:33:38.792864 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:33:38.792856    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/179468b3-b372-4905-87d4-1de35ed0ce2e-tmp-dir\") pod \"e2e-lifecycle-model-d91d15cb-kserve-7d4b9fb6f9-85t64\" (UID: \"179468b3-b372-4905-87d4-1de35ed0ce2e\") " pod="ai-tenant-e2e-mt-d91d15cb/e2e-lifecycle-model-d91d15cb-kserve-7d4b9fb6f9-85t64"
Jul 12 17:33:38.793032 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:33:38.792899    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/179468b3-b372-4905-87d4-1de35ed0ce2e-model-cache\") pod \"e2e-lifecycle-model-d91d15cb-kserve-7d4b9fb6f9-85t64\" (UID: \"179468b3-b372-4905-87d4-1de35ed0ce2e\") " pod="ai-tenant-e2e-mt-d91d15cb/e2e-lifecycle-model-d91d15cb-kserve-7d4b9fb6f9-85t64"
Jul 12 17:33:38.794649 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:33:38.794628    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/179468b3-b372-4905-87d4-1de35ed0ce2e-dshm\") pod \"e2e-lifecycle-model-d91d15cb-kserve-7d4b9fb6f9-85t64\" (UID: \"179468b3-b372-4905-87d4-1de35ed0ce2e\") " pod="ai-tenant-e2e-mt-d91d15cb/e2e-lifecycle-model-d91d15cb-kserve-7d4b9fb6f9-85t64"
Jul 12 17:33:38.794890 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:33:38.794875    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/179468b3-b372-4905-87d4-1de35ed0ce2e-tls-certs\") pod \"e2e-lifecycle-model-d91d15cb-kserve-7d4b9fb6f9-85t64\" (UID: \"179468b3-b372-4905-87d4-1de35ed0ce2e\") " pod="ai-tenant-e2e-mt-d91d15cb/e2e-lifecycle-model-d91d15cb-kserve-7d4b9fb6f9-85t64"
Jul 12 17:33:38.808301 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:33:38.808282    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8852\" (UniqueName: \"kubernetes.io/projected/179468b3-b372-4905-87d4-1de35ed0ce2e-kube-api-access-p8852\") pod \"e2e-lifecycle-model-d91d15cb-kserve-7d4b9fb6f9-85t64\" (UID: \"179468b3-b372-4905-87d4-1de35ed0ce2e\") " pod="ai-tenant-e2e-mt-d91d15cb/e2e-lifecycle-model-d91d15cb-kserve-7d4b9fb6f9-85t64"
Jul 12 17:33:38.908143 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:33:38.908088    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-d91d15cb/e2e-lifecycle-model-d91d15cb-kserve-7d4b9fb6f9-85t64"
Jul 12 17:33:39.034223 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:33:39.034191    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-d91d15cb/e2e-lifecycle-model-d91d15cb-kserve-7d4b9fb6f9-85t64"]
Jul 12 17:33:39.037286 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:33:39.037255    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod179468b3_b372_4905_87d4_1de35ed0ce2e.slice/crio-d25a8b041da5a8a024e6e8355e898bc435f3324352de3c5ab8ebab6f8931507f WatchSource:0}: Error finding container d25a8b041da5a8a024e6e8355e898bc435f3324352de3c5ab8ebab6f8931507f: Status 404 returned error can't find the container with id d25a8b041da5a8a024e6e8355e898bc435f3324352de3c5ab8ebab6f8931507f
Jul 12 17:33:39.976294 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:33:39.976257    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-d91d15cb/e2e-lifecycle-model-d91d15cb-kserve-7d4b9fb6f9-85t64" event={"ID":"179468b3-b372-4905-87d4-1de35ed0ce2e","Type":"ContainerStarted","Data":"7557f23d0656ca4b0ef309c6d6361ec7234dce6b19e9e4be805e90cdc31deddd"}
Jul 12 17:33:39.976294 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:33:39.976299    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-d91d15cb/e2e-lifecycle-model-d91d15cb-kserve-7d4b9fb6f9-85t64" event={"ID":"179468b3-b372-4905-87d4-1de35ed0ce2e","Type":"ContainerStarted","Data":"d25a8b041da5a8a024e6e8355e898bc435f3324352de3c5ab8ebab6f8931507f"}
Jul 12 17:33:39.991922 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:33:39.991879    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-d91d15cb/e2e-lifecycle-model-d91d15cb-kserve-7d4b9fb6f9-85t64" podStartSLOduration=1.9918643569999999 podStartE2EDuration="1.991864357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 17:33:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 17:33:39.990909349 +0000 UTC m=+1828.111226145" watchObservedRunningTime="2026-07-12 17:33:39.991864357 +0000 UTC m=+1828.112181152"
Jul 12 17:33:43.987288 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:33:43.987253    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl4zn"]
Jul 12 17:33:48.908853 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:33:48.908819    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-d91d15cb/e2e-lifecycle-model-d91d15cb-kserve-7d4b9fb6f9-85t64"
Jul 12 17:33:48.909271 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:33:48.908896    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-d91d15cb/e2e-lifecycle-model-d91d15cb-kserve-7d4b9fb6f9-85t64"
Jul 12 17:33:48.920787 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:33:48.920759    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-d91d15cb/e2e-lifecycle-model-d91d15cb-kserve-7d4b9fb6f9-85t64"
Jul 12 17:33:49.025685 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:33:49.025660    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-d91d15cb/e2e-lifecycle-model-d91d15cb-kserve-7d4b9fb6f9-85t64"
Jul 12 17:34:03.392564 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:34:03.392524    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl4zn"]
Jul 12 17:37:24.786229 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:37:24.786195    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl4zn"]
Jul 12 17:37:30.858627 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:37:30.858597    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-d91d15cb/e2e-lifecycle-model-d91d15cb-kserve-7d4b9fb6f9-85t64"]
Jul 12 17:37:45.879325 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:37:45.879234    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-d91d15cb/e2e-lifecycle-model-d91d15cb-kserve-7d4b9fb6f9-85t64" podUID="179468b3-b372-4905-87d4-1de35ed0ce2e" containerName="main" containerID="cri-o://7557f23d0656ca4b0ef309c6d6361ec7234dce6b19e9e4be805e90cdc31deddd" gracePeriod=45
Jul 12 17:37:46.124558 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:37:46.124532    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-d91d15cb/e2e-lifecycle-model-d91d15cb-kserve-7d4b9fb6f9-85t64"
Jul 12 17:37:46.295584 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:37:46.295551    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/179468b3-b372-4905-87d4-1de35ed0ce2e-tmp-dir\") pod \"179468b3-b372-4905-87d4-1de35ed0ce2e\" (UID: \"179468b3-b372-4905-87d4-1de35ed0ce2e\") "
Jul 12 17:37:46.295762 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:37:46.295629    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p8852\" (UniqueName: \"kubernetes.io/projected/179468b3-b372-4905-87d4-1de35ed0ce2e-kube-api-access-p8852\") pod \"179468b3-b372-4905-87d4-1de35ed0ce2e\" (UID: \"179468b3-b372-4905-87d4-1de35ed0ce2e\") "
Jul 12 17:37:46.295762 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:37:46.295666    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/179468b3-b372-4905-87d4-1de35ed0ce2e-dshm\") pod \"179468b3-b372-4905-87d4-1de35ed0ce2e\" (UID: \"179468b3-b372-4905-87d4-1de35ed0ce2e\") "
Jul 12 17:37:46.295762 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:37:46.295685    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/179468b3-b372-4905-87d4-1de35ed0ce2e-tls-certs\") pod \"179468b3-b372-4905-87d4-1de35ed0ce2e\" (UID: \"179468b3-b372-4905-87d4-1de35ed0ce2e\") "
Jul 12 17:37:46.295762 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:37:46.295728    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/179468b3-b372-4905-87d4-1de35ed0ce2e-home\") pod \"179468b3-b372-4905-87d4-1de35ed0ce2e\" (UID: \"179468b3-b372-4905-87d4-1de35ed0ce2e\") "
Jul 12 17:37:46.295762 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:37:46.295748    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/179468b3-b372-4905-87d4-1de35ed0ce2e-model-cache\") pod \"179468b3-b372-4905-87d4-1de35ed0ce2e\" (UID: \"179468b3-b372-4905-87d4-1de35ed0ce2e\") "
Jul 12 17:37:46.296034 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:37:46.295834    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/179468b3-b372-4905-87d4-1de35ed0ce2e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "179468b3-b372-4905-87d4-1de35ed0ce2e" (UID: "179468b3-b372-4905-87d4-1de35ed0ce2e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 12 17:37:46.296104 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:37:46.296042    2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/179468b3-b372-4905-87d4-1de35ed0ce2e-tmp-dir\") on node \"ip-10-0-141-94.ec2.internal\" DevicePath \"\""
Jul 12 17:37:46.296104 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:37:46.296083    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/179468b3-b372-4905-87d4-1de35ed0ce2e-model-cache" (OuterVolumeSpecName: "model-cache") pod "179468b3-b372-4905-87d4-1de35ed0ce2e" (UID: "179468b3-b372-4905-87d4-1de35ed0ce2e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 12 17:37:46.296104 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:37:46.296093    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/179468b3-b372-4905-87d4-1de35ed0ce2e-home" (OuterVolumeSpecName: "home") pod "179468b3-b372-4905-87d4-1de35ed0ce2e" (UID: "179468b3-b372-4905-87d4-1de35ed0ce2e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 12 17:37:46.297878 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:37:46.297834    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/179468b3-b372-4905-87d4-1de35ed0ce2e-dshm" (OuterVolumeSpecName: "dshm") pod "179468b3-b372-4905-87d4-1de35ed0ce2e" (UID: "179468b3-b372-4905-87d4-1de35ed0ce2e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 12 17:37:46.297976 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:37:46.297891    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/179468b3-b372-4905-87d4-1de35ed0ce2e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "179468b3-b372-4905-87d4-1de35ed0ce2e" (UID: "179468b3-b372-4905-87d4-1de35ed0ce2e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 12 17:37:46.297976 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:37:46.297917    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/179468b3-b372-4905-87d4-1de35ed0ce2e-kube-api-access-p8852" (OuterVolumeSpecName: "kube-api-access-p8852") pod "179468b3-b372-4905-87d4-1de35ed0ce2e" (UID: "179468b3-b372-4905-87d4-1de35ed0ce2e"). InnerVolumeSpecName "kube-api-access-p8852". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 12 17:37:46.396825 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:37:46.396794    2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/179468b3-b372-4905-87d4-1de35ed0ce2e-model-cache\") on node \"ip-10-0-141-94.ec2.internal\" DevicePath \"\""
Jul 12 17:37:46.396825 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:37:46.396821    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8852\" (UniqueName: \"kubernetes.io/projected/179468b3-b372-4905-87d4-1de35ed0ce2e-kube-api-access-p8852\") on node \"ip-10-0-141-94.ec2.internal\" DevicePath \"\""
Jul 12 17:37:46.396825 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:37:46.396830    2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/179468b3-b372-4905-87d4-1de35ed0ce2e-dshm\") on node \"ip-10-0-141-94.ec2.internal\" DevicePath \"\""
Jul 12 17:37:46.397003 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:37:46.396839    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/179468b3-b372-4905-87d4-1de35ed0ce2e-tls-certs\") on node \"ip-10-0-141-94.ec2.internal\" DevicePath \"\""
Jul 12 17:37:46.397003 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:37:46.396848    2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/179468b3-b372-4905-87d4-1de35ed0ce2e-home\") on node \"ip-10-0-141-94.ec2.internal\" DevicePath \"\""
Jul 12 17:37:46.982870 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:37:46.982835    2569 generic.go:356] "Generic (PLEG): container finished" podID="179468b3-b372-4905-87d4-1de35ed0ce2e" containerID="7557f23d0656ca4b0ef309c6d6361ec7234dce6b19e9e4be805e90cdc31deddd" exitCode=0
Jul 12 17:37:46.983281 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:37:46.982879    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-d91d15cb/e2e-lifecycle-model-d91d15cb-kserve-7d4b9fb6f9-85t64"
Jul 12 17:37:46.983281 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:37:46.982911    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-d91d15cb/e2e-lifecycle-model-d91d15cb-kserve-7d4b9fb6f9-85t64" event={"ID":"179468b3-b372-4905-87d4-1de35ed0ce2e","Type":"ContainerDied","Data":"7557f23d0656ca4b0ef309c6d6361ec7234dce6b19e9e4be805e90cdc31deddd"}
Jul 12 17:37:46.983281 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:37:46.982943    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-d91d15cb/e2e-lifecycle-model-d91d15cb-kserve-7d4b9fb6f9-85t64" event={"ID":"179468b3-b372-4905-87d4-1de35ed0ce2e","Type":"ContainerDied","Data":"d25a8b041da5a8a024e6e8355e898bc435f3324352de3c5ab8ebab6f8931507f"}
Jul 12 17:37:46.983281 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:37:46.982960    2569 scope.go:117] "RemoveContainer" containerID="7557f23d0656ca4b0ef309c6d6361ec7234dce6b19e9e4be805e90cdc31deddd"
Jul 12 17:37:46.991097 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:37:46.991073    2569 scope.go:117] "RemoveContainer" containerID="7557f23d0656ca4b0ef309c6d6361ec7234dce6b19e9e4be805e90cdc31deddd"
Jul 12 17:37:46.991327 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:37:46.991308    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7557f23d0656ca4b0ef309c6d6361ec7234dce6b19e9e4be805e90cdc31deddd\": container with ID starting with 7557f23d0656ca4b0ef309c6d6361ec7234dce6b19e9e4be805e90cdc31deddd not found: ID does not exist" containerID="7557f23d0656ca4b0ef309c6d6361ec7234dce6b19e9e4be805e90cdc31deddd"
Jul 12 17:37:46.991393 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:37:46.991338    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7557f23d0656ca4b0ef309c6d6361ec7234dce6b19e9e4be805e90cdc31deddd"} err="failed to get container status \"7557f23d0656ca4b0ef309c6d6361ec7234dce6b19e9e4be805e90cdc31deddd\": rpc error: code = NotFound desc = could not find container \"7557f23d0656ca4b0ef309c6d6361ec7234dce6b19e9e4be805e90cdc31deddd\": container with ID starting with 7557f23d0656ca4b0ef309c6d6361ec7234dce6b19e9e4be805e90cdc31deddd not found: ID does not exist"
Jul 12 17:37:46.998211 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:37:46.998188    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-d91d15cb/e2e-lifecycle-model-d91d15cb-kserve-7d4b9fb6f9-85t64"]
Jul 12 17:37:47.001666 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:37:47.001643    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-d91d15cb/e2e-lifecycle-model-d91d15cb-kserve-7d4b9fb6f9-85t64"]
Jul 12 17:37:48.379361 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:37:48.379324    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="179468b3-b372-4905-87d4-1de35ed0ce2e" path="/var/lib/kubelet/pods/179468b3-b372-4905-87d4-1de35ed0ce2e/volumes"
Jul 12 17:38:12.411786 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:38:12.411747    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9xbrj_6b618491-4e81-4708-ab1b-2781d1df7c87/ovn-acl-logging/0.log"
Jul 12 17:38:12.417191 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:38:12.417168    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9xbrj_6b618491-4e81-4708-ab1b-2781d1df7c87/ovn-acl-logging/0.log"
Jul 12 17:38:21.494715 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:38:21.494681    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl4zn"]
Jul 12 17:38:34.887843 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:38:34.887809    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl4zn"]
Jul 12 17:39:23.082404 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:39:23.082325    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl4zn"]
Jul 12 17:39:40.496031 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:39:40.495981    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl4zn"]
Jul 12 17:39:49.986981 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:39:49.986943    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-c5fd66/test-model-c5fd66-kserve-cbf54747c-bgnrr"]
Jul 12 17:39:49.987379 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:39:49.987245    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="179468b3-b372-4905-87d4-1de35ed0ce2e" containerName="main"
Jul 12 17:39:49.987379 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:39:49.987257    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="179468b3-b372-4905-87d4-1de35ed0ce2e" containerName="main"
Jul 12 17:39:49.987379 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:39:49.987333    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="179468b3-b372-4905-87d4-1de35ed0ce2e" containerName="main"
Jul 12 17:39:49.990147 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:39:49.990131    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c5fd66/test-model-c5fd66-kserve-cbf54747c-bgnrr"
Jul 12 17:39:49.993476 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:39:49.993449    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-c5fd66\"/\"openshift-service-ca.crt\""
Jul 12 17:39:49.993476 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:39:49.993468    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-c5fd66\"/\"kube-root-ca.crt\""
Jul 12 17:39:49.993685 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:39:49.993535    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-c5fd66\"/\"default-dockercfg-kgcz9\""
Jul 12 17:39:49.993685 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:39:49.993556    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-c5fd66\"/\"test-model-c5fd66-kserve-self-signed-certs\""
Jul 12 17:39:49.998219 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:39:49.998195    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-c5fd66/test-model-c5fd66-kserve-cbf54747c-bgnrr"]
Jul 12 17:39:50.006328 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:39:50.006300    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f351e9d-560d-473b-861e-428ee4ff399f-home\") pod \"test-model-c5fd66-kserve-cbf54747c-bgnrr\" (UID: \"0f351e9d-560d-473b-861e-428ee4ff399f\") " pod="ai-tenant-e2e-inf-a-c5fd66/test-model-c5fd66-kserve-cbf54747c-bgnrr"
Jul 12 17:39:50.006568 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:39:50.006348    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f351e9d-560d-473b-861e-428ee4ff399f-tmp-dir\") pod \"test-model-c5fd66-kserve-cbf54747c-bgnrr\" (UID: \"0f351e9d-560d-473b-861e-428ee4ff399f\") " pod="ai-tenant-e2e-inf-a-c5fd66/test-model-c5fd66-kserve-cbf54747c-bgnrr"
Jul 12 17:39:50.006568 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:39:50.006447    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f351e9d-560d-473b-861e-428ee4ff399f-tls-certs\") pod \"test-model-c5fd66-kserve-cbf54747c-bgnrr\" (UID: \"0f351e9d-560d-473b-861e-428ee4ff399f\") " pod="ai-tenant-e2e-inf-a-c5fd66/test-model-c5fd66-kserve-cbf54747c-bgnrr"
Jul 12 17:39:50.006568 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:39:50.006562    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f351e9d-560d-473b-861e-428ee4ff399f-dshm\") pod \"test-model-c5fd66-kserve-cbf54747c-bgnrr\" (UID: \"0f351e9d-560d-473b-861e-428ee4ff399f\") " pod="ai-tenant-e2e-inf-a-c5fd66/test-model-c5fd66-kserve-cbf54747c-bgnrr"
Jul 12 17:39:50.006752 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:39:50.006637    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xkhr\" (UniqueName: \"kubernetes.io/projected/0f351e9d-560d-473b-861e-428ee4ff399f-kube-api-access-7xkhr\") pod \"test-model-c5fd66-kserve-cbf54747c-bgnrr\" (UID: \"0f351e9d-560d-473b-861e-428ee4ff399f\") " pod="ai-tenant-e2e-inf-a-c5fd66/test-model-c5fd66-kserve-cbf54747c-bgnrr"
Jul 12 17:39:50.006752 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:39:50.006681    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f351e9d-560d-473b-861e-428ee4ff399f-model-cache\") pod \"test-model-c5fd66-kserve-cbf54747c-bgnrr\" (UID: \"0f351e9d-560d-473b-861e-428ee4ff399f\") " pod="ai-tenant-e2e-inf-a-c5fd66/test-model-c5fd66-kserve-cbf54747c-bgnrr"
Jul 12 17:39:50.107957 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:39:50.107907    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f351e9d-560d-473b-861e-428ee4ff399f-tmp-dir\") pod \"test-model-c5fd66-kserve-cbf54747c-bgnrr\" (UID: \"0f351e9d-560d-473b-861e-428ee4ff399f\") " pod="ai-tenant-e2e-inf-a-c5fd66/test-model-c5fd66-kserve-cbf54747c-bgnrr"
Jul 12 17:39:50.107957 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:39:50.107961    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f351e9d-560d-473b-861e-428ee4ff399f-tls-certs\") pod \"test-model-c5fd66-kserve-cbf54747c-bgnrr\" (UID: \"0f351e9d-560d-473b-861e-428ee4ff399f\") " pod="ai-tenant-e2e-inf-a-c5fd66/test-model-c5fd66-kserve-cbf54747c-bgnrr"
Jul 12 17:39:50.108211 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:39:50.108010    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f351e9d-560d-473b-861e-428ee4ff399f-dshm\") pod \"test-model-c5fd66-kserve-cbf54747c-bgnrr\" (UID: \"0f351e9d-560d-473b-861e-428ee4ff399f\") " pod="ai-tenant-e2e-inf-a-c5fd66/test-model-c5fd66-kserve-cbf54747c-bgnrr"
Jul 12 17:39:50.108211 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:39:50.108042    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xkhr\" (UniqueName: \"kubernetes.io/projected/0f351e9d-560d-473b-861e-428ee4ff399f-kube-api-access-7xkhr\") pod \"test-model-c5fd66-kserve-cbf54747c-bgnrr\" (UID: \"0f351e9d-560d-473b-861e-428ee4ff399f\") " pod="ai-tenant-e2e-inf-a-c5fd66/test-model-c5fd66-kserve-cbf54747c-bgnrr"
Jul 12 17:39:50.108211 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:39:50.108073    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f351e9d-560d-473b-861e-428ee4ff399f-model-cache\") pod \"test-model-c5fd66-kserve-cbf54747c-bgnrr\" (UID: \"0f351e9d-560d-473b-861e-428ee4ff399f\") " pod="ai-tenant-e2e-inf-a-c5fd66/test-model-c5fd66-kserve-cbf54747c-bgnrr"
Jul 12 17:39:50.108211 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:39:50.108125    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f351e9d-560d-473b-861e-428ee4ff399f-home\") pod \"test-model-c5fd66-kserve-cbf54747c-bgnrr\" (UID: \"0f351e9d-560d-473b-861e-428ee4ff399f\") " pod="ai-tenant-e2e-inf-a-c5fd66/test-model-c5fd66-kserve-cbf54747c-bgnrr"
Jul 12 17:39:50.108413 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:39:50.108379    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f351e9d-560d-473b-861e-428ee4ff399f-tmp-dir\") pod \"test-model-c5fd66-kserve-cbf54747c-bgnrr\" (UID: \"0f351e9d-560d-473b-861e-428ee4ff399f\") " pod="ai-tenant-e2e-inf-a-c5fd66/test-model-c5fd66-kserve-cbf54747c-bgnrr"
Jul 12 17:39:50.108468 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:39:50.108450    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f351e9d-560d-473b-861e-428ee4ff399f-home\") pod \"test-model-c5fd66-kserve-cbf54747c-bgnrr\" (UID: \"0f351e9d-560d-473b-861e-428ee4ff399f\") " pod="ai-tenant-e2e-inf-a-c5fd66/test-model-c5fd66-kserve-cbf54747c-bgnrr"
Jul 12 17:39:50.108572 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:39:50.108484    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f351e9d-560d-473b-861e-428ee4ff399f-model-cache\") pod \"test-model-c5fd66-kserve-cbf54747c-bgnrr\" (UID: \"0f351e9d-560d-473b-861e-428ee4ff399f\") " pod="ai-tenant-e2e-inf-a-c5fd66/test-model-c5fd66-kserve-cbf54747c-bgnrr"
Jul 12 17:39:50.110284 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:39:50.110257    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f351e9d-560d-473b-861e-428ee4ff399f-dshm\") pod \"test-model-c5fd66-kserve-cbf54747c-bgnrr\" (UID: \"0f351e9d-560d-473b-861e-428ee4ff399f\") " pod="ai-tenant-e2e-inf-a-c5fd66/test-model-c5fd66-kserve-cbf54747c-bgnrr"
Jul 12 17:39:50.110537 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:39:50.110500    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f351e9d-560d-473b-861e-428ee4ff399f-tls-certs\") pod \"test-model-c5fd66-kserve-cbf54747c-bgnrr\" (UID: \"0f351e9d-560d-473b-861e-428ee4ff399f\") " pod="ai-tenant-e2e-inf-a-c5fd66/test-model-c5fd66-kserve-cbf54747c-bgnrr"
Jul 12 17:39:50.115749 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:39:50.115730    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xkhr\" (UniqueName: \"kubernetes.io/projected/0f351e9d-560d-473b-861e-428ee4ff399f-kube-api-access-7xkhr\") pod \"test-model-c5fd66-kserve-cbf54747c-bgnrr\" (UID: \"0f351e9d-560d-473b-861e-428ee4ff399f\") " pod="ai-tenant-e2e-inf-a-c5fd66/test-model-c5fd66-kserve-cbf54747c-bgnrr"
Jul 12 17:39:50.300442 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:39:50.300351    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c5fd66/test-model-c5fd66-kserve-cbf54747c-bgnrr"
Jul 12 17:39:50.434800 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:39:50.434759    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-c5fd66/test-model-c5fd66-kserve-cbf54747c-bgnrr"]
Jul 12 17:39:50.437479 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:39:50.437453    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0f351e9d_560d_473b_861e_428ee4ff399f.slice/crio-a8a2b03765b5aa8fd7c232d9261c338afa242e0870be0aff62fee7ff05695da0 WatchSource:0}: Error finding container a8a2b03765b5aa8fd7c232d9261c338afa242e0870be0aff62fee7ff05695da0: Status 404 returned error can't find the container with id a8a2b03765b5aa8fd7c232d9261c338afa242e0870be0aff62fee7ff05695da0
Jul 12 17:39:50.492479 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:39:50.492450    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c5fd66/test-model-c5fd66-kserve-cbf54747c-bgnrr" event={"ID":"0f351e9d-560d-473b-861e-428ee4ff399f","Type":"ContainerStarted","Data":"a8a2b03765b5aa8fd7c232d9261c338afa242e0870be0aff62fee7ff05695da0"}
Jul 12 17:39:51.498422 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:39:51.498383    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c5fd66/test-model-c5fd66-kserve-cbf54747c-bgnrr" event={"ID":"0f351e9d-560d-473b-861e-428ee4ff399f","Type":"ContainerStarted","Data":"10db83e9ac5e7fd65ceb2d635324d6c17d527f431fc770e427fb9f9b9fe67c99"}
Jul 12 17:39:51.513791 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:39:51.513736    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-c5fd66/test-model-c5fd66-kserve-cbf54747c-bgnrr" podStartSLOduration=2.513719524 podStartE2EDuration="2.513719524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 17:39:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 17:39:51.512942928 +0000 UTC m=+2199.633259727" watchObservedRunningTime="2026-07-12 17:39:51.513719524 +0000 UTC m=+2199.634036300"
Jul 12 17:39:53.089642 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:39:53.089599    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl4zn"]
Jul 12 17:40:00.301389 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:40:00.301343    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-c5fd66/test-model-c5fd66-kserve-cbf54747c-bgnrr"
Jul 12 17:40:00.301777 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:40:00.301486    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-c5fd66/test-model-c5fd66-kserve-cbf54747c-bgnrr"
Jul 12 17:40:00.313854 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:40:00.313829    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-c5fd66/test-model-c5fd66-kserve-cbf54747c-bgnrr"
Jul 12 17:40:00.549607 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:40:00.549579    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-c5fd66/test-model-c5fd66-kserve-cbf54747c-bgnrr"
Jul 12 17:40:07.696913 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:40:07.696878    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl4zn"]
Jul 12 17:40:15.888521 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:40:15.888475    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-7fa4a5/test-model-7fa4a5-kserve-56b9b4c4cc-6dljc"]
Jul 12 17:40:15.891969 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:40:15.891947    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-7fa4a5/test-model-7fa4a5-kserve-56b9b4c4cc-6dljc"
Jul 12 17:40:15.895339 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:40:15.895312    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-7fa4a5\"/\"openshift-service-ca.crt\""
Jul 12 17:40:15.895339 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:40:15.895324    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-7fa4a5\"/\"default-dockercfg-psh7x\""
Jul 12 17:40:15.895541 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:40:15.895326    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-7fa4a5\"/\"test-model-7fa4a5-kserve-self-signed-certs\""
Jul 12 17:40:15.895541 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:40:15.895374    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-7fa4a5\"/\"kube-root-ca.crt\""
Jul 12 17:40:15.902866 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:40:15.902840    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-7fa4a5/test-model-7fa4a5-kserve-56b9b4c4cc-6dljc"]
Jul 12 17:40:16.049259 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:40:16.049223    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e8ce9f9-eb0a-427c-9fee-7cc55af7c599-model-cache\") pod \"test-model-7fa4a5-kserve-56b9b4c4cc-6dljc\" (UID: \"3e8ce9f9-eb0a-427c-9fee-7cc55af7c599\") " pod="ai-tenant-e2e-inf-b-7fa4a5/test-model-7fa4a5-kserve-56b9b4c4cc-6dljc"
Jul 12 17:40:16.049470 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:40:16.049293    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e8ce9f9-eb0a-427c-9fee-7cc55af7c599-dshm\") pod \"test-model-7fa4a5-kserve-56b9b4c4cc-6dljc\" (UID: \"3e8ce9f9-eb0a-427c-9fee-7cc55af7c599\") " pod="ai-tenant-e2e-inf-b-7fa4a5/test-model-7fa4a5-kserve-56b9b4c4cc-6dljc"
Jul 12 17:40:16.049470 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:40:16.049338    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e8ce9f9-eb0a-427c-9fee-7cc55af7c599-tls-certs\") pod \"test-model-7fa4a5-kserve-56b9b4c4cc-6dljc\" (UID: \"3e8ce9f9-eb0a-427c-9fee-7cc55af7c599\") " pod="ai-tenant-e2e-inf-b-7fa4a5/test-model-7fa4a5-kserve-56b9b4c4cc-6dljc"
Jul 12 17:40:16.049470 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:40:16.049386    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e8ce9f9-eb0a-427c-9fee-7cc55af7c599-home\") pod \"test-model-7fa4a5-kserve-56b9b4c4cc-6dljc\" (UID: \"3e8ce9f9-eb0a-427c-9fee-7cc55af7c599\") " pod="ai-tenant-e2e-inf-b-7fa4a5/test-model-7fa4a5-kserve-56b9b4c4cc-6dljc"
Jul 12 17:40:16.049470 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:40:16.049408    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fgps\" (UniqueName: \"kubernetes.io/projected/3e8ce9f9-eb0a-427c-9fee-7cc55af7c599-kube-api-access-4fgps\") pod \"test-model-7fa4a5-kserve-56b9b4c4cc-6dljc\" (UID: \"3e8ce9f9-eb0a-427c-9fee-7cc55af7c599\") " pod="ai-tenant-e2e-inf-b-7fa4a5/test-model-7fa4a5-kserve-56b9b4c4cc-6dljc"
Jul 12 17:40:16.049470 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:40:16.049451    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e8ce9f9-eb0a-427c-9fee-7cc55af7c599-tmp-dir\") pod \"test-model-7fa4a5-kserve-56b9b4c4cc-6dljc\" (UID: \"3e8ce9f9-eb0a-427c-9fee-7cc55af7c599\") " pod="ai-tenant-e2e-inf-b-7fa4a5/test-model-7fa4a5-kserve-56b9b4c4cc-6dljc"
Jul 12 17:40:16.150601 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:40:16.150486    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e8ce9f9-eb0a-427c-9fee-7cc55af7c599-home\") pod \"test-model-7fa4a5-kserve-56b9b4c4cc-6dljc\" (UID: \"3e8ce9f9-eb0a-427c-9fee-7cc55af7c599\") " pod="ai-tenant-e2e-inf-b-7fa4a5/test-model-7fa4a5-kserve-56b9b4c4cc-6dljc"
Jul 12 17:40:16.150601 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:40:16.150544    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fgps\" (UniqueName: \"kubernetes.io/projected/3e8ce9f9-eb0a-427c-9fee-7cc55af7c599-kube-api-access-4fgps\") pod \"test-model-7fa4a5-kserve-56b9b4c4cc-6dljc\" (UID: \"3e8ce9f9-eb0a-427c-9fee-7cc55af7c599\") " pod="ai-tenant-e2e-inf-b-7fa4a5/test-model-7fa4a5-kserve-56b9b4c4cc-6dljc"
Jul 12 17:40:16.150601 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:40:16.150577    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e8ce9f9-eb0a-427c-9fee-7cc55af7c599-tmp-dir\") pod \"test-model-7fa4a5-kserve-56b9b4c4cc-6dljc\" (UID: \"3e8ce9f9-eb0a-427c-9fee-7cc55af7c599\") " pod="ai-tenant-e2e-inf-b-7fa4a5/test-model-7fa4a5-kserve-56b9b4c4cc-6dljc"
Jul 12 17:40:16.150879 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:40:16.150607    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e8ce9f9-eb0a-427c-9fee-7cc55af7c599-model-cache\") pod \"test-model-7fa4a5-kserve-56b9b4c4cc-6dljc\" (UID: \"3e8ce9f9-eb0a-427c-9fee-7cc55af7c599\") " pod="ai-tenant-e2e-inf-b-7fa4a5/test-model-7fa4a5-kserve-56b9b4c4cc-6dljc"
Jul 12 17:40:16.150879 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:40:16.150659    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e8ce9f9-eb0a-427c-9fee-7cc55af7c599-dshm\") pod \"test-model-7fa4a5-kserve-56b9b4c4cc-6dljc\" (UID: \"3e8ce9f9-eb0a-427c-9fee-7cc55af7c599\") " pod="ai-tenant-e2e-inf-b-7fa4a5/test-model-7fa4a5-kserve-56b9b4c4cc-6dljc"
Jul 12 17:40:16.150879 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:40:16.150803    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e8ce9f9-eb0a-427c-9fee-7cc55af7c599-tls-certs\") pod \"test-model-7fa4a5-kserve-56b9b4c4cc-6dljc\" (UID: \"3e8ce9f9-eb0a-427c-9fee-7cc55af7c599\") " pod="ai-tenant-e2e-inf-b-7fa4a5/test-model-7fa4a5-kserve-56b9b4c4cc-6dljc"
Jul 12 17:40:16.151010 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:40:16.150953    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e8ce9f9-eb0a-427c-9fee-7cc55af7c599-home\") pod \"test-model-7fa4a5-kserve-56b9b4c4cc-6dljc\" (UID: \"3e8ce9f9-eb0a-427c-9fee-7cc55af7c599\") " pod="ai-tenant-e2e-inf-b-7fa4a5/test-model-7fa4a5-kserve-56b9b4c4cc-6dljc"
Jul 12 17:40:16.151010 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:40:16.150982    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e8ce9f9-eb0a-427c-9fee-7cc55af7c599-tmp-dir\") pod \"test-model-7fa4a5-kserve-56b9b4c4cc-6dljc\" (UID: \"3e8ce9f9-eb0a-427c-9fee-7cc55af7c599\") " pod="ai-tenant-e2e-inf-b-7fa4a5/test-model-7fa4a5-kserve-56b9b4c4cc-6dljc"
Jul 12 17:40:16.151086 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:40:16.151010    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e8ce9f9-eb0a-427c-9fee-7cc55af7c599-model-cache\") pod \"test-model-7fa4a5-kserve-56b9b4c4cc-6dljc\" (UID: \"3e8ce9f9-eb0a-427c-9fee-7cc55af7c599\") " pod="ai-tenant-e2e-inf-b-7fa4a5/test-model-7fa4a5-kserve-56b9b4c4cc-6dljc"
Jul 12 17:40:16.153023 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:40:16.152993    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e8ce9f9-eb0a-427c-9fee-7cc55af7c599-dshm\") pod \"test-model-7fa4a5-kserve-56b9b4c4cc-6dljc\" (UID: \"3e8ce9f9-eb0a-427c-9fee-7cc55af7c599\") " pod="ai-tenant-e2e-inf-b-7fa4a5/test-model-7fa4a5-kserve-56b9b4c4cc-6dljc"
Jul 12 17:40:16.153199 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:40:16.153179    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e8ce9f9-eb0a-427c-9fee-7cc55af7c599-tls-certs\") pod \"test-model-7fa4a5-kserve-56b9b4c4cc-6dljc\" (UID: \"3e8ce9f9-eb0a-427c-9fee-7cc55af7c599\") " pod="ai-tenant-e2e-inf-b-7fa4a5/test-model-7fa4a5-kserve-56b9b4c4cc-6dljc"
Jul 12 17:40:16.161086 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:40:16.161064    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fgps\" (UniqueName: \"kubernetes.io/projected/3e8ce9f9-eb0a-427c-9fee-7cc55af7c599-kube-api-access-4fgps\") pod \"test-model-7fa4a5-kserve-56b9b4c4cc-6dljc\" (UID: \"3e8ce9f9-eb0a-427c-9fee-7cc55af7c599\") " pod="ai-tenant-e2e-inf-b-7fa4a5/test-model-7fa4a5-kserve-56b9b4c4cc-6dljc"
Jul 12 17:40:16.202184 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:40:16.202144    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-7fa4a5/test-model-7fa4a5-kserve-56b9b4c4cc-6dljc"
Jul 12 17:40:16.330170 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:40:16.330132    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-7fa4a5/test-model-7fa4a5-kserve-56b9b4c4cc-6dljc"]
Jul 12 17:40:16.334439 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:40:16.334412    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3e8ce9f9_eb0a_427c_9fee_7cc55af7c599.slice/crio-454a5a3a2bc7481110f3abbe51ee360610c573b256afa236bd0663cb57151f81 WatchSource:0}: Error finding container 454a5a3a2bc7481110f3abbe51ee360610c573b256afa236bd0663cb57151f81: Status 404 returned error can't find the container with id 454a5a3a2bc7481110f3abbe51ee360610c573b256afa236bd0663cb57151f81
Jul 12 17:40:16.606884 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:40:16.606849    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-7fa4a5/test-model-7fa4a5-kserve-56b9b4c4cc-6dljc" event={"ID":"3e8ce9f9-eb0a-427c-9fee-7cc55af7c599","Type":"ContainerStarted","Data":"3be359da825a11ac848ca6e725230af6259c45e8fb9b2ecf9927d6ef15b197c9"}
Jul 12 17:40:16.606884 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:40:16.606889    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-7fa4a5/test-model-7fa4a5-kserve-56b9b4c4cc-6dljc" event={"ID":"3e8ce9f9-eb0a-427c-9fee-7cc55af7c599","Type":"ContainerStarted","Data":"454a5a3a2bc7481110f3abbe51ee360610c573b256afa236bd0663cb57151f81"}
Jul 12 17:40:16.623729 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:40:16.623679    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-7fa4a5/test-model-7fa4a5-kserve-56b9b4c4cc-6dljc" podStartSLOduration=1.623664438 podStartE2EDuration="1.623664438s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 17:40:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 17:40:16.62216963 +0000 UTC m=+2224.742486429" watchObservedRunningTime="2026-07-12 17:40:16.623664438 +0000 UTC m=+2224.743981235"
Jul 12 17:40:24.425733 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:40:24.425702    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl4zn"]
Jul 12 17:40:26.202925 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:40:26.202886    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-7fa4a5/test-model-7fa4a5-kserve-56b9b4c4cc-6dljc"
Jul 12 17:40:26.203379 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:40:26.202946    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-7fa4a5/test-model-7fa4a5-kserve-56b9b4c4cc-6dljc"
Jul 12 17:40:26.215277 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:40:26.215251    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-7fa4a5/test-model-7fa4a5-kserve-56b9b4c4cc-6dljc"
Jul 12 17:40:26.660626 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:40:26.660548    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-7fa4a5/test-model-7fa4a5-kserve-56b9b4c4cc-6dljc"
Jul 12 17:40:36.423871 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:40:36.423784    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-c5fd66/test-model-c5fd66-kserve-cbf54747c-bgnrr"]
Jul 12 17:40:37.496620 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:40:37.496582    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-7fa4a5/test-model-7fa4a5-kserve-56b9b4c4cc-6dljc"]
Jul 12 17:40:43.083790 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:40:43.083748    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl4zn"]
Jul 12 17:40:51.444297 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:40:51.444252    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-c5fd66/test-model-c5fd66-kserve-cbf54747c-bgnrr" podUID="0f351e9d-560d-473b-861e-428ee4ff399f" containerName="main" containerID="cri-o://10db83e9ac5e7fd65ceb2d635324d6c17d527f431fc770e427fb9f9b9fe67c99" gracePeriod=45
Jul 12 17:40:51.688422 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:40:51.688394    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c5fd66/test-model-c5fd66-kserve-cbf54747c-bgnrr"
Jul 12 17:40:51.754880 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:40:51.754795    2569 generic.go:356] "Generic (PLEG): container finished" podID="0f351e9d-560d-473b-861e-428ee4ff399f" containerID="10db83e9ac5e7fd65ceb2d635324d6c17d527f431fc770e427fb9f9b9fe67c99" exitCode=0
Jul 12 17:40:51.754880 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:40:51.754845    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c5fd66/test-model-c5fd66-kserve-cbf54747c-bgnrr"
Jul 12 17:40:51.755065 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:40:51.754884    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c5fd66/test-model-c5fd66-kserve-cbf54747c-bgnrr" event={"ID":"0f351e9d-560d-473b-861e-428ee4ff399f","Type":"ContainerDied","Data":"10db83e9ac5e7fd65ceb2d635324d6c17d527f431fc770e427fb9f9b9fe67c99"}
Jul 12 17:40:51.755065 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:40:51.754928    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c5fd66/test-model-c5fd66-kserve-cbf54747c-bgnrr" event={"ID":"0f351e9d-560d-473b-861e-428ee4ff399f","Type":"ContainerDied","Data":"a8a2b03765b5aa8fd7c232d9261c338afa242e0870be0aff62fee7ff05695da0"}
Jul 12 17:40:51.755065 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:40:51.754947    2569 scope.go:117] "RemoveContainer" containerID="10db83e9ac5e7fd65ceb2d635324d6c17d527f431fc770e427fb9f9b9fe67c99"
Jul 12 17:40:51.763013 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:40:51.762994    2569 scope.go:117] "RemoveContainer" containerID="10db83e9ac5e7fd65ceb2d635324d6c17d527f431fc770e427fb9f9b9fe67c99"
Jul 12 17:40:51.763270 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:40:51.763253    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10db83e9ac5e7fd65ceb2d635324d6c17d527f431fc770e427fb9f9b9fe67c99\": container with ID starting with 10db83e9ac5e7fd65ceb2d635324d6c17d527f431fc770e427fb9f9b9fe67c99 not found: ID does not exist" containerID="10db83e9ac5e7fd65ceb2d635324d6c17d527f431fc770e427fb9f9b9fe67c99"
Jul 12 17:40:51.763318 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:40:51.763278    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10db83e9ac5e7fd65ceb2d635324d6c17d527f431fc770e427fb9f9b9fe67c99"} err="failed to get container status \"10db83e9ac5e7fd65ceb2d635324d6c17d527f431fc770e427fb9f9b9fe67c99\": rpc error: code = NotFound desc = could not find container \"10db83e9ac5e7fd65ceb2d635324d6c17d527f431fc770e427fb9f9b9fe67c99\": container with ID starting with 10db83e9ac5e7fd65ceb2d635324d6c17d527f431fc770e427fb9f9b9fe67c99 not found: ID does not exist"
Jul 12 17:40:51.781655 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:40:51.781628    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f351e9d-560d-473b-861e-428ee4ff399f-home\") pod \"0f351e9d-560d-473b-861e-428ee4ff399f\" (UID: \"0f351e9d-560d-473b-861e-428ee4ff399f\") "
Jul 12 17:40:51.781748 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:40:51.781663    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f351e9d-560d-473b-861e-428ee4ff399f-tmp-dir\") pod \"0f351e9d-560d-473b-861e-428ee4ff399f\" (UID: \"0f351e9d-560d-473b-861e-428ee4ff399f\") "
Jul 12 17:40:51.781748 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:40:51.781696    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f351e9d-560d-473b-861e-428ee4ff399f-tls-certs\") pod \"0f351e9d-560d-473b-861e-428ee4ff399f\" (UID: \"0f351e9d-560d-473b-861e-428ee4ff399f\") "
Jul 12 17:40:51.781748 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:40:51.781713    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7xkhr\" (UniqueName: \"kubernetes.io/projected/0f351e9d-560d-473b-861e-428ee4ff399f-kube-api-access-7xkhr\") pod \"0f351e9d-560d-473b-861e-428ee4ff399f\" (UID: \"0f351e9d-560d-473b-861e-428ee4ff399f\") "
Jul 12 17:40:51.781943 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:40:51.781758    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f351e9d-560d-473b-861e-428ee4ff399f-dshm\") pod \"0f351e9d-560d-473b-861e-428ee4ff399f\" (UID: \"0f351e9d-560d-473b-861e-428ee4ff399f\") "
Jul 12 17:40:51.781943 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:40:51.781772    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f351e9d-560d-473b-861e-428ee4ff399f-model-cache\") pod \"0f351e9d-560d-473b-861e-428ee4ff399f\" (UID: \"0f351e9d-560d-473b-861e-428ee4ff399f\") "
Jul 12 17:40:51.782052 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:40:51.781939    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f351e9d-560d-473b-861e-428ee4ff399f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0f351e9d-560d-473b-861e-428ee4ff399f" (UID: "0f351e9d-560d-473b-861e-428ee4ff399f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 12 17:40:51.782052 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:40:51.781949    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f351e9d-560d-473b-861e-428ee4ff399f-home" (OuterVolumeSpecName: "home") pod "0f351e9d-560d-473b-861e-428ee4ff399f" (UID: "0f351e9d-560d-473b-861e-428ee4ff399f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 12 17:40:51.782152 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:40:51.782086    2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f351e9d-560d-473b-861e-428ee4ff399f-home\") on node \"ip-10-0-141-94.ec2.internal\" DevicePath \"\""
Jul 12 17:40:51.782152 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:40:51.782107    2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f351e9d-560d-473b-861e-428ee4ff399f-tmp-dir\") on node \"ip-10-0-141-94.ec2.internal\" DevicePath \"\""
Jul 12 17:40:51.782152 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:40:51.782103    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f351e9d-560d-473b-861e-428ee4ff399f-model-cache" (OuterVolumeSpecName: "model-cache") pod "0f351e9d-560d-473b-861e-428ee4ff399f" (UID: "0f351e9d-560d-473b-861e-428ee4ff399f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 12 17:40:51.783867 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:40:51.783841    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f351e9d-560d-473b-861e-428ee4ff399f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0f351e9d-560d-473b-861e-428ee4ff399f" (UID: "0f351e9d-560d-473b-861e-428ee4ff399f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 12 17:40:51.783970 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:40:51.783943    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f351e9d-560d-473b-861e-428ee4ff399f-dshm" (OuterVolumeSpecName: "dshm") pod "0f351e9d-560d-473b-861e-428ee4ff399f" (UID: "0f351e9d-560d-473b-861e-428ee4ff399f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 12 17:40:51.784036 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:40:51.783976    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f351e9d-560d-473b-861e-428ee4ff399f-kube-api-access-7xkhr" (OuterVolumeSpecName: "kube-api-access-7xkhr") pod "0f351e9d-560d-473b-861e-428ee4ff399f" (UID: "0f351e9d-560d-473b-861e-428ee4ff399f"). InnerVolumeSpecName "kube-api-access-7xkhr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 12 17:40:51.883062 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:40:51.883020    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f351e9d-560d-473b-861e-428ee4ff399f-tls-certs\") on node \"ip-10-0-141-94.ec2.internal\" DevicePath \"\""
Jul 12 17:40:51.883062 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:40:51.883054    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xkhr\" (UniqueName: \"kubernetes.io/projected/0f351e9d-560d-473b-861e-428ee4ff399f-kube-api-access-7xkhr\") on node \"ip-10-0-141-94.ec2.internal\" DevicePath \"\""
Jul 12 17:40:51.883062 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:40:51.883068    2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f351e9d-560d-473b-861e-428ee4ff399f-dshm\") on node \"ip-10-0-141-94.ec2.internal\" DevicePath \"\""
Jul 12 17:40:51.883294 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:40:51.883080    2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f351e9d-560d-473b-861e-428ee4ff399f-model-cache\") on node \"ip-10-0-141-94.ec2.internal\" DevicePath \"\""
Jul 12 17:40:52.080010 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:40:52.079964    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-c5fd66/test-model-c5fd66-kserve-cbf54747c-bgnrr"]
Jul 12 17:40:52.086236 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:40:52.086211    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-c5fd66/test-model-c5fd66-kserve-cbf54747c-bgnrr"]
Jul 12 17:40:52.379634 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:40:52.379558    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f351e9d-560d-473b-861e-428ee4ff399f" path="/var/lib/kubelet/pods/0f351e9d-560d-473b-861e-428ee4ff399f/volumes"
Jul 12 17:40:52.516666 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:40:52.516628    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-7fa4a5/test-model-7fa4a5-kserve-56b9b4c4cc-6dljc" podUID="3e8ce9f9-eb0a-427c-9fee-7cc55af7c599" containerName="main" containerID="cri-o://3be359da825a11ac848ca6e725230af6259c45e8fb9b2ecf9927d6ef15b197c9" gracePeriod=45
Jul 12 17:40:52.762848 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:40:52.762817    2569 generic.go:356] "Generic (PLEG): container finished" podID="3e8ce9f9-eb0a-427c-9fee-7cc55af7c599" containerID="3be359da825a11ac848ca6e725230af6259c45e8fb9b2ecf9927d6ef15b197c9" exitCode=0
Jul 12 17:40:52.762967 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:40:52.762868    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-7fa4a5/test-model-7fa4a5-kserve-56b9b4c4cc-6dljc" event={"ID":"3e8ce9f9-eb0a-427c-9fee-7cc55af7c599","Type":"ContainerDied","Data":"3be359da825a11ac848ca6e725230af6259c45e8fb9b2ecf9927d6ef15b197c9"}
Jul 12 17:40:52.762967 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:40:52.762899    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-7fa4a5/test-model-7fa4a5-kserve-56b9b4c4cc-6dljc" event={"ID":"3e8ce9f9-eb0a-427c-9fee-7cc55af7c599","Type":"ContainerDied","Data":"454a5a3a2bc7481110f3abbe51ee360610c573b256afa236bd0663cb57151f81"}
Jul 12 17:40:52.762967 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:40:52.762911    2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="454a5a3a2bc7481110f3abbe51ee360610c573b256afa236bd0663cb57151f81"
Jul 12 17:40:52.765584 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:40:52.765568    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-7fa4a5/test-model-7fa4a5-kserve-56b9b4c4cc-6dljc"
Jul 12 17:40:52.891315 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:40:52.891251    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e8ce9f9-eb0a-427c-9fee-7cc55af7c599-tmp-dir\") pod \"3e8ce9f9-eb0a-427c-9fee-7cc55af7c599\" (UID: \"3e8ce9f9-eb0a-427c-9fee-7cc55af7c599\") "
Jul 12 17:40:52.891315 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:40:52.891296    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e8ce9f9-eb0a-427c-9fee-7cc55af7c599-home\") pod \"3e8ce9f9-eb0a-427c-9fee-7cc55af7c599\" (UID: \"3e8ce9f9-eb0a-427c-9fee-7cc55af7c599\") "
Jul 12 17:40:52.891491 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:40:52.891320    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e8ce9f9-eb0a-427c-9fee-7cc55af7c599-tls-certs\") pod \"3e8ce9f9-eb0a-427c-9fee-7cc55af7c599\" (UID: \"3e8ce9f9-eb0a-427c-9fee-7cc55af7c599\") "
Jul 12 17:40:52.891491 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:40:52.891368    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4fgps\" (UniqueName: \"kubernetes.io/projected/3e8ce9f9-eb0a-427c-9fee-7cc55af7c599-kube-api-access-4fgps\") pod \"3e8ce9f9-eb0a-427c-9fee-7cc55af7c599\" (UID: \"3e8ce9f9-eb0a-427c-9fee-7cc55af7c599\") "
Jul 12 17:40:52.891491 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:40:52.891396    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e8ce9f9-eb0a-427c-9fee-7cc55af7c599-dshm\") pod \"3e8ce9f9-eb0a-427c-9fee-7cc55af7c599\" (UID: \"3e8ce9f9-eb0a-427c-9fee-7cc55af7c599\") "
Jul 12 17:40:52.891491 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:40:52.891420    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e8ce9f9-eb0a-427c-9fee-7cc55af7c599-model-cache\") pod \"3e8ce9f9-eb0a-427c-9fee-7cc55af7c599\" (UID: \"3e8ce9f9-eb0a-427c-9fee-7cc55af7c599\") "
Jul 12 17:40:52.891721 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:40:52.891573    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e8ce9f9-eb0a-427c-9fee-7cc55af7c599-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3e8ce9f9-eb0a-427c-9fee-7cc55af7c599" (UID: "3e8ce9f9-eb0a-427c-9fee-7cc55af7c599"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 12 17:40:52.891721 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:40:52.891581    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e8ce9f9-eb0a-427c-9fee-7cc55af7c599-home" (OuterVolumeSpecName: "home") pod "3e8ce9f9-eb0a-427c-9fee-7cc55af7c599" (UID: "3e8ce9f9-eb0a-427c-9fee-7cc55af7c599"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 12 17:40:52.891721 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:40:52.891714    2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e8ce9f9-eb0a-427c-9fee-7cc55af7c599-tmp-dir\") on node \"ip-10-0-141-94.ec2.internal\" DevicePath \"\""
Jul 12 17:40:52.891876 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:40:52.891734    2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e8ce9f9-eb0a-427c-9fee-7cc55af7c599-home\") on node \"ip-10-0-141-94.ec2.internal\" DevicePath \"\""
Jul 12 17:40:52.891876 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:40:52.891742    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e8ce9f9-eb0a-427c-9fee-7cc55af7c599-model-cache" (OuterVolumeSpecName: "model-cache") pod "3e8ce9f9-eb0a-427c-9fee-7cc55af7c599" (UID: "3e8ce9f9-eb0a-427c-9fee-7cc55af7c599"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 12 17:40:52.893545 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:40:52.893525    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e8ce9f9-eb0a-427c-9fee-7cc55af7c599-dshm" (OuterVolumeSpecName: "dshm") pod "3e8ce9f9-eb0a-427c-9fee-7cc55af7c599" (UID: "3e8ce9f9-eb0a-427c-9fee-7cc55af7c599"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 12 17:40:52.893966 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:40:52.893944    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e8ce9f9-eb0a-427c-9fee-7cc55af7c599-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3e8ce9f9-eb0a-427c-9fee-7cc55af7c599" (UID: "3e8ce9f9-eb0a-427c-9fee-7cc55af7c599"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 12 17:40:52.893966 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:40:52.893958    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e8ce9f9-eb0a-427c-9fee-7cc55af7c599-kube-api-access-4fgps" (OuterVolumeSpecName: "kube-api-access-4fgps") pod "3e8ce9f9-eb0a-427c-9fee-7cc55af7c599" (UID: "3e8ce9f9-eb0a-427c-9fee-7cc55af7c599"). InnerVolumeSpecName "kube-api-access-4fgps". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 12 17:40:52.992878 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:40:52.992857    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fgps\" (UniqueName: \"kubernetes.io/projected/3e8ce9f9-eb0a-427c-9fee-7cc55af7c599-kube-api-access-4fgps\") on node \"ip-10-0-141-94.ec2.internal\" DevicePath \"\""
Jul 12 17:40:52.992878 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:40:52.992877    2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e8ce9f9-eb0a-427c-9fee-7cc55af7c599-dshm\") on node \"ip-10-0-141-94.ec2.internal\" DevicePath \"\""
Jul 12 17:40:52.993010 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:40:52.992886    2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e8ce9f9-eb0a-427c-9fee-7cc55af7c599-model-cache\") on node \"ip-10-0-141-94.ec2.internal\" DevicePath \"\""
Jul 12 17:40:52.993010 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:40:52.992895    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e8ce9f9-eb0a-427c-9fee-7cc55af7c599-tls-certs\") on node \"ip-10-0-141-94.ec2.internal\" DevicePath \"\""
Jul 12 17:40:53.766832 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:40:53.766800    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-7fa4a5/test-model-7fa4a5-kserve-56b9b4c4cc-6dljc"
Jul 12 17:40:53.787696 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:40:53.787665    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-7fa4a5/test-model-7fa4a5-kserve-56b9b4c4cc-6dljc"]
Jul 12 17:40:53.789166 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:40:53.789143    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-7fa4a5/test-model-7fa4a5-kserve-56b9b4c4cc-6dljc"]
Jul 12 17:40:54.378810 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:40:54.378781    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3e8ce9f9-eb0a-427c-9fee-7cc55af7c599" path="/var/lib/kubelet/pods/3e8ce9f9-eb0a-427c-9fee-7cc55af7c599/volumes"
Jul 12 17:40:58.190899 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:40:58.190868    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl4zn"]
Jul 12 17:41:17.284536 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:41:17.284482    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl4zn"]
Jul 12 17:41:38.390811 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:41:38.390777    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl4zn"]
Jul 12 17:41:55.698815 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:41:55.698781    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl4zn"]
Jul 12 17:42:07.888401 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:42:07.888316    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl4zn"]
Jul 12 17:42:33.391795 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:42:33.391755    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl4zn"]
Jul 12 17:42:55.589432 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:42:55.589399    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl4zn"]
Jul 12 17:43:11.791899 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:43:11.791868    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl4zn"]
Jul 12 17:43:12.444551 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:43:12.444500    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9xbrj_6b618491-4e81-4708-ab1b-2781d1df7c87/ovn-acl-logging/0.log"
Jul 12 17:43:12.449114 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:43:12.449094    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9xbrj_6b618491-4e81-4708-ab1b-2781d1df7c87/ovn-acl-logging/0.log"
Jul 12 17:43:22.497396 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:43:22.497364    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-e767e8/auth-test-model-e767e8-kserve-8847d9cc9-gzf7n"]
Jul 12 17:43:22.497765 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:43:22.497693    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f351e9d-560d-473b-861e-428ee4ff399f" containerName="main"
Jul 12 17:43:22.497765 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:43:22.497705    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f351e9d-560d-473b-861e-428ee4ff399f" containerName="main"
Jul 12 17:43:22.497765 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:43:22.497716    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e8ce9f9-eb0a-427c-9fee-7cc55af7c599" containerName="main"
Jul 12 17:43:22.497765 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:43:22.497723    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e8ce9f9-eb0a-427c-9fee-7cc55af7c599" containerName="main"
Jul 12 17:43:22.497892 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:43:22.497776    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e8ce9f9-eb0a-427c-9fee-7cc55af7c599" containerName="main"
Jul 12 17:43:22.497892 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:43:22.497785    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f351e9d-560d-473b-861e-428ee4ff399f" containerName="main"
Jul 12 17:43:22.500535 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:43:22.500502    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-e767e8/auth-test-model-e767e8-kserve-8847d9cc9-gzf7n"
Jul 12 17:43:22.503460 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:43:22.503435    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-e767e8\"/\"openshift-service-ca.crt\""
Jul 12 17:43:22.503460 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:43:22.503449    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-e767e8\"/\"auth-test-model-e767e8-kserve-self-signed-certs\""
Jul 12 17:43:22.503659 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:43:22.503454    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-e767e8\"/\"kube-root-ca.crt\""
Jul 12 17:43:22.503659 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:43:22.503566    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-e767e8\"/\"default-dockercfg-qd8cl\""
Jul 12 17:43:22.508193 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:43:22.508174    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-e767e8/auth-test-model-e767e8-kserve-8847d9cc9-gzf7n"]
Jul 12 17:43:22.578665 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:43:22.578637    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc9cd027-7643-454f-a778-fdd4a29e5442-tmp-dir\") pod \"auth-test-model-e767e8-kserve-8847d9cc9-gzf7n\" (UID: \"cc9cd027-7643-454f-a778-fdd4a29e5442\") " pod="ai-tenant-e2e-shared-a-e767e8/auth-test-model-e767e8-kserve-8847d9cc9-gzf7n"
Jul 12 17:43:22.578807 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:43:22.578699    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc9cd027-7643-454f-a778-fdd4a29e5442-tls-certs\") pod \"auth-test-model-e767e8-kserve-8847d9cc9-gzf7n\" (UID: \"cc9cd027-7643-454f-a778-fdd4a29e5442\") " pod="ai-tenant-e2e-shared-a-e767e8/auth-test-model-e767e8-kserve-8847d9cc9-gzf7n"
Jul 12 17:43:22.578807 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:43:22.578737    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc9cd027-7643-454f-a778-fdd4a29e5442-dshm\") pod \"auth-test-model-e767e8-kserve-8847d9cc9-gzf7n\" (UID: \"cc9cd027-7643-454f-a778-fdd4a29e5442\") " pod="ai-tenant-e2e-shared-a-e767e8/auth-test-model-e767e8-kserve-8847d9cc9-gzf7n"
Jul 12 17:43:22.578807 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:43:22.578753    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc9cd027-7643-454f-a778-fdd4a29e5442-home\") pod \"auth-test-model-e767e8-kserve-8847d9cc9-gzf7n\" (UID: \"cc9cd027-7643-454f-a778-fdd4a29e5442\") " pod="ai-tenant-e2e-shared-a-e767e8/auth-test-model-e767e8-kserve-8847d9cc9-gzf7n"
Jul 12 17:43:22.578807 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:43:22.578777    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc9cd027-7643-454f-a778-fdd4a29e5442-model-cache\") pod \"auth-test-model-e767e8-kserve-8847d9cc9-gzf7n\" (UID: \"cc9cd027-7643-454f-a778-fdd4a29e5442\") " pod="ai-tenant-e2e-shared-a-e767e8/auth-test-model-e767e8-kserve-8847d9cc9-gzf7n"
Jul 12 17:43:22.578807 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:43:22.578793    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxxv4\" (UniqueName: \"kubernetes.io/projected/cc9cd027-7643-454f-a778-fdd4a29e5442-kube-api-access-bxxv4\") pod \"auth-test-model-e767e8-kserve-8847d9cc9-gzf7n\" (UID: \"cc9cd027-7643-454f-a778-fdd4a29e5442\") " pod="ai-tenant-e2e-shared-a-e767e8/auth-test-model-e767e8-kserve-8847d9cc9-gzf7n"
Jul 12 17:43:22.680019 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:43:22.679980    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc9cd027-7643-454f-a778-fdd4a29e5442-tls-certs\") pod \"auth-test-model-e767e8-kserve-8847d9cc9-gzf7n\" (UID: \"cc9cd027-7643-454f-a778-fdd4a29e5442\") " pod="ai-tenant-e2e-shared-a-e767e8/auth-test-model-e767e8-kserve-8847d9cc9-gzf7n"
Jul 12 17:43:22.680175 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:43:22.680033    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc9cd027-7643-454f-a778-fdd4a29e5442-dshm\") pod \"auth-test-model-e767e8-kserve-8847d9cc9-gzf7n\" (UID: \"cc9cd027-7643-454f-a778-fdd4a29e5442\") " pod="ai-tenant-e2e-shared-a-e767e8/auth-test-model-e767e8-kserve-8847d9cc9-gzf7n"
Jul 12 17:43:22.680175 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:43:22.680070    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc9cd027-7643-454f-a778-fdd4a29e5442-home\") pod \"auth-test-model-e767e8-kserve-8847d9cc9-gzf7n\" (UID: \"cc9cd027-7643-454f-a778-fdd4a29e5442\") " pod="ai-tenant-e2e-shared-a-e767e8/auth-test-model-e767e8-kserve-8847d9cc9-gzf7n"
Jul 12 17:43:22.680175 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:43:22.680088    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc9cd027-7643-454f-a778-fdd4a29e5442-model-cache\") pod \"auth-test-model-e767e8-kserve-8847d9cc9-gzf7n\" (UID: \"cc9cd027-7643-454f-a778-fdd4a29e5442\") " pod="ai-tenant-e2e-shared-a-e767e8/auth-test-model-e767e8-kserve-8847d9cc9-gzf7n"
Jul 12 17:43:22.680175 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:43:22.680104    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxxv4\" (UniqueName: \"kubernetes.io/projected/cc9cd027-7643-454f-a778-fdd4a29e5442-kube-api-access-bxxv4\") pod \"auth-test-model-e767e8-kserve-8847d9cc9-gzf7n\" (UID: \"cc9cd027-7643-454f-a778-fdd4a29e5442\") " pod="ai-tenant-e2e-shared-a-e767e8/auth-test-model-e767e8-kserve-8847d9cc9-gzf7n"
Jul 12 17:43:22.680175 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:43:22.680133    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc9cd027-7643-454f-a778-fdd4a29e5442-tmp-dir\") pod \"auth-test-model-e767e8-kserve-8847d9cc9-gzf7n\" (UID: \"cc9cd027-7643-454f-a778-fdd4a29e5442\") " pod="ai-tenant-e2e-shared-a-e767e8/auth-test-model-e767e8-kserve-8847d9cc9-gzf7n"
Jul 12 17:43:22.680503 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:43:22.680477    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc9cd027-7643-454f-a778-fdd4a29e5442-home\") pod \"auth-test-model-e767e8-kserve-8847d9cc9-gzf7n\" (UID: \"cc9cd027-7643-454f-a778-fdd4a29e5442\") " pod="ai-tenant-e2e-shared-a-e767e8/auth-test-model-e767e8-kserve-8847d9cc9-gzf7n"
Jul 12 17:43:22.680607 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:43:22.680549    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc9cd027-7643-454f-a778-fdd4a29e5442-tmp-dir\") pod \"auth-test-model-e767e8-kserve-8847d9cc9-gzf7n\" (UID: \"cc9cd027-7643-454f-a778-fdd4a29e5442\") " pod="ai-tenant-e2e-shared-a-e767e8/auth-test-model-e767e8-kserve-8847d9cc9-gzf7n"
Jul 12 17:43:22.680665 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:43:22.680637    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc9cd027-7643-454f-a778-fdd4a29e5442-model-cache\") pod \"auth-test-model-e767e8-kserve-8847d9cc9-gzf7n\" (UID: \"cc9cd027-7643-454f-a778-fdd4a29e5442\") " pod="ai-tenant-e2e-shared-a-e767e8/auth-test-model-e767e8-kserve-8847d9cc9-gzf7n"
Jul 12 17:43:22.682320 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:43:22.682300    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc9cd027-7643-454f-a778-fdd4a29e5442-dshm\") pod \"auth-test-model-e767e8-kserve-8847d9cc9-gzf7n\" (UID: \"cc9cd027-7643-454f-a778-fdd4a29e5442\") " pod="ai-tenant-e2e-shared-a-e767e8/auth-test-model-e767e8-kserve-8847d9cc9-gzf7n"
Jul 12 17:43:22.682663 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:43:22.682646    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc9cd027-7643-454f-a778-fdd4a29e5442-tls-certs\") pod \"auth-test-model-e767e8-kserve-8847d9cc9-gzf7n\" (UID: \"cc9cd027-7643-454f-a778-fdd4a29e5442\") " pod="ai-tenant-e2e-shared-a-e767e8/auth-test-model-e767e8-kserve-8847d9cc9-gzf7n"
Jul 12 17:43:22.687370 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:43:22.687347    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxxv4\" (UniqueName: \"kubernetes.io/projected/cc9cd027-7643-454f-a778-fdd4a29e5442-kube-api-access-bxxv4\") pod \"auth-test-model-e767e8-kserve-8847d9cc9-gzf7n\" (UID: \"cc9cd027-7643-454f-a778-fdd4a29e5442\") " pod="ai-tenant-e2e-shared-a-e767e8/auth-test-model-e767e8-kserve-8847d9cc9-gzf7n"
Jul 12 17:43:22.809752 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:43:22.809689    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-e767e8/auth-test-model-e767e8-kserve-8847d9cc9-gzf7n"
Jul 12 17:43:22.934874 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:43:22.934842    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-e767e8/auth-test-model-e767e8-kserve-8847d9cc9-gzf7n"]
Jul 12 17:43:22.937784 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:43:22.937750    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcc9cd027_7643_454f_a778_fdd4a29e5442.slice/crio-10439010b0cf283bfc48b18a9b0623029c38e15490dbd73dbdce1a5013e3b500 WatchSource:0}: Error finding container 10439010b0cf283bfc48b18a9b0623029c38e15490dbd73dbdce1a5013e3b500: Status 404 returned error can't find the container with id 10439010b0cf283bfc48b18a9b0623029c38e15490dbd73dbdce1a5013e3b500
Jul 12 17:43:23.389161 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:43:23.389121    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-e767e8/auth-test-model-e767e8-kserve-8847d9cc9-gzf7n" event={"ID":"cc9cd027-7643-454f-a778-fdd4a29e5442","Type":"ContainerStarted","Data":"072add77746c5b1844627013ec162d700a858787a8f8be1eabcbe2f2d9b621dd"}
Jul 12 17:43:23.389161 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:43:23.389162    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-e767e8/auth-test-model-e767e8-kserve-8847d9cc9-gzf7n" event={"ID":"cc9cd027-7643-454f-a778-fdd4a29e5442","Type":"ContainerStarted","Data":"10439010b0cf283bfc48b18a9b0623029c38e15490dbd73dbdce1a5013e3b500"}
Jul 12 17:43:23.404968 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:43:23.404923    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-e767e8/auth-test-model-e767e8-kserve-8847d9cc9-gzf7n" podStartSLOduration=1.404910302 podStartE2EDuration="1.404910302s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 17:43:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 17:43:23.404458227 +0000 UTC m=+2411.524775024" watchObservedRunningTime="2026-07-12 17:43:23.404910302 +0000 UTC m=+2411.525227098"
Jul 12 17:43:26.288970 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:43:26.288933    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl4zn"]
Jul 12 17:43:32.809776 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:43:32.809744    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-e767e8/auth-test-model-e767e8-kserve-8847d9cc9-gzf7n"
Jul 12 17:43:32.810254 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:43:32.809812    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-e767e8/auth-test-model-e767e8-kserve-8847d9cc9-gzf7n"
Jul 12 17:43:32.822011 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:43:32.821989    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-e767e8/auth-test-model-e767e8-kserve-8847d9cc9-gzf7n"
Jul 12 17:43:33.439173 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:43:33.439141    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-e767e8/auth-test-model-e767e8-kserve-8847d9cc9-gzf7n"
Jul 12 17:43:56.191825 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:43:56.191738    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-9c75b6/auth-test-model-9c75b6-kserve-5bdc976c55-vbzwg"]
Jul 12 17:43:56.195149 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:43:56.195127    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-9c75b6/auth-test-model-9c75b6-kserve-5bdc976c55-vbzwg"
Jul 12 17:43:56.197559 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:43:56.197536    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-9c75b6\"/\"default-dockercfg-n24tv\""
Jul 12 17:43:56.197559 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:43:56.197556    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-9c75b6\"/\"kube-root-ca.crt\""
Jul 12 17:43:56.198544 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:43:56.198524    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-9c75b6\"/\"auth-test-model-9c75b6-kserve-self-signed-certs\""
Jul 12 17:43:56.198647 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:43:56.198575    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-9c75b6\"/\"openshift-service-ca.crt\""
Jul 12 17:43:56.204538 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:43:56.204499    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-9c75b6/auth-test-model-9c75b6-kserve-5bdc976c55-vbzwg"]
Jul 12 17:43:56.346911 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:43:56.346853    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/352994a7-d5cc-4932-8aee-f0ed1f173dbb-model-cache\") pod \"auth-test-model-9c75b6-kserve-5bdc976c55-vbzwg\" (UID: \"352994a7-d5cc-4932-8aee-f0ed1f173dbb\") " pod="ai-tenant-e2e-shared-b-9c75b6/auth-test-model-9c75b6-kserve-5bdc976c55-vbzwg"
Jul 12 17:43:56.347149 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:43:56.346939    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/352994a7-d5cc-4932-8aee-f0ed1f173dbb-dshm\") pod \"auth-test-model-9c75b6-kserve-5bdc976c55-vbzwg\" (UID: \"352994a7-d5cc-4932-8aee-f0ed1f173dbb\") " pod="ai-tenant-e2e-shared-b-9c75b6/auth-test-model-9c75b6-kserve-5bdc976c55-vbzwg"
Jul 12 17:43:56.347149 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:43:56.346985    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/352994a7-d5cc-4932-8aee-f0ed1f173dbb-tls-certs\") pod \"auth-test-model-9c75b6-kserve-5bdc976c55-vbzwg\" (UID: \"352994a7-d5cc-4932-8aee-f0ed1f173dbb\") " pod="ai-tenant-e2e-shared-b-9c75b6/auth-test-model-9c75b6-kserve-5bdc976c55-vbzwg"
Jul 12 17:43:56.347149 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:43:56.347012    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/352994a7-d5cc-4932-8aee-f0ed1f173dbb-home\") pod \"auth-test-model-9c75b6-kserve-5bdc976c55-vbzwg\" (UID: \"352994a7-d5cc-4932-8aee-f0ed1f173dbb\") " pod="ai-tenant-e2e-shared-b-9c75b6/auth-test-model-9c75b6-kserve-5bdc976c55-vbzwg"
Jul 12 17:43:56.347149 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:43:56.347045    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/352994a7-d5cc-4932-8aee-f0ed1f173dbb-tmp-dir\") pod \"auth-test-model-9c75b6-kserve-5bdc976c55-vbzwg\" (UID: \"352994a7-d5cc-4932-8aee-f0ed1f173dbb\") " pod="ai-tenant-e2e-shared-b-9c75b6/auth-test-model-9c75b6-kserve-5bdc976c55-vbzwg"
Jul 12 17:43:56.347149 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:43:56.347135    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwz4b\" (UniqueName: \"kubernetes.io/projected/352994a7-d5cc-4932-8aee-f0ed1f173dbb-kube-api-access-rwz4b\") pod \"auth-test-model-9c75b6-kserve-5bdc976c55-vbzwg\" (UID: \"352994a7-d5cc-4932-8aee-f0ed1f173dbb\") " pod="ai-tenant-e2e-shared-b-9c75b6/auth-test-model-9c75b6-kserve-5bdc976c55-vbzwg"
Jul 12 17:43:56.447998 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:43:56.447907    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/352994a7-d5cc-4932-8aee-f0ed1f173dbb-model-cache\") pod \"auth-test-model-9c75b6-kserve-5bdc976c55-vbzwg\" (UID: \"352994a7-d5cc-4932-8aee-f0ed1f173dbb\") " pod="ai-tenant-e2e-shared-b-9c75b6/auth-test-model-9c75b6-kserve-5bdc976c55-vbzwg"
Jul 12 17:43:56.447998 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:43:56.447965    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/352994a7-d5cc-4932-8aee-f0ed1f173dbb-dshm\") pod \"auth-test-model-9c75b6-kserve-5bdc976c55-vbzwg\" (UID: \"352994a7-d5cc-4932-8aee-f0ed1f173dbb\") " pod="ai-tenant-e2e-shared-b-9c75b6/auth-test-model-9c75b6-kserve-5bdc976c55-vbzwg"
Jul 12 17:43:56.448265 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:43:56.448164    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/352994a7-d5cc-4932-8aee-f0ed1f173dbb-tls-certs\") pod \"auth-test-model-9c75b6-kserve-5bdc976c55-vbzwg\" (UID: \"352994a7-d5cc-4932-8aee-f0ed1f173dbb\") " pod="ai-tenant-e2e-shared-b-9c75b6/auth-test-model-9c75b6-kserve-5bdc976c55-vbzwg"
Jul 12 17:43:56.448265 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:43:56.448208    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/352994a7-d5cc-4932-8aee-f0ed1f173dbb-home\") pod \"auth-test-model-9c75b6-kserve-5bdc976c55-vbzwg\" (UID: \"352994a7-d5cc-4932-8aee-f0ed1f173dbb\") " pod="ai-tenant-e2e-shared-b-9c75b6/auth-test-model-9c75b6-kserve-5bdc976c55-vbzwg"
Jul 12 17:43:56.448265 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:43:56.448257    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/352994a7-d5cc-4932-8aee-f0ed1f173dbb-tmp-dir\") pod \"auth-test-model-9c75b6-kserve-5bdc976c55-vbzwg\" (UID: \"352994a7-d5cc-4932-8aee-f0ed1f173dbb\") " pod="ai-tenant-e2e-shared-b-9c75b6/auth-test-model-9c75b6-kserve-5bdc976c55-vbzwg"
Jul 12 17:43:56.448432 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:43:56.448332    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwz4b\" (UniqueName: \"kubernetes.io/projected/352994a7-d5cc-4932-8aee-f0ed1f173dbb-kube-api-access-rwz4b\") pod \"auth-test-model-9c75b6-kserve-5bdc976c55-vbzwg\" (UID: \"352994a7-d5cc-4932-8aee-f0ed1f173dbb\") " pod="ai-tenant-e2e-shared-b-9c75b6/auth-test-model-9c75b6-kserve-5bdc976c55-vbzwg"
Jul 12 17:43:56.448488 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:43:56.448432    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/352994a7-d5cc-4932-8aee-f0ed1f173dbb-model-cache\") pod \"auth-test-model-9c75b6-kserve-5bdc976c55-vbzwg\" (UID: \"352994a7-d5cc-4932-8aee-f0ed1f173dbb\") " pod="ai-tenant-e2e-shared-b-9c75b6/auth-test-model-9c75b6-kserve-5bdc976c55-vbzwg"
Jul 12 17:43:56.448581 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:43:56.448549    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/352994a7-d5cc-4932-8aee-f0ed1f173dbb-home\") pod \"auth-test-model-9c75b6-kserve-5bdc976c55-vbzwg\" (UID: \"352994a7-d5cc-4932-8aee-f0ed1f173dbb\") " pod="ai-tenant-e2e-shared-b-9c75b6/auth-test-model-9c75b6-kserve-5bdc976c55-vbzwg"
Jul 12 17:43:56.448687 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:43:56.448667    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/352994a7-d5cc-4932-8aee-f0ed1f173dbb-tmp-dir\") pod \"auth-test-model-9c75b6-kserve-5bdc976c55-vbzwg\" (UID: \"352994a7-d5cc-4932-8aee-f0ed1f173dbb\") " pod="ai-tenant-e2e-shared-b-9c75b6/auth-test-model-9c75b6-kserve-5bdc976c55-vbzwg"
Jul 12 17:43:56.450325 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:43:56.450305    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/352994a7-d5cc-4932-8aee-f0ed1f173dbb-dshm\") pod \"auth-test-model-9c75b6-kserve-5bdc976c55-vbzwg\" (UID: \"352994a7-d5cc-4932-8aee-f0ed1f173dbb\") " pod="ai-tenant-e2e-shared-b-9c75b6/auth-test-model-9c75b6-kserve-5bdc976c55-vbzwg"
Jul 12 17:43:56.450628 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:43:56.450608    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/352994a7-d5cc-4932-8aee-f0ed1f173dbb-tls-certs\") pod \"auth-test-model-9c75b6-kserve-5bdc976c55-vbzwg\" (UID: \"352994a7-d5cc-4932-8aee-f0ed1f173dbb\") " pod="ai-tenant-e2e-shared-b-9c75b6/auth-test-model-9c75b6-kserve-5bdc976c55-vbzwg"
Jul 12 17:43:56.455887 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:43:56.455868    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwz4b\" (UniqueName: \"kubernetes.io/projected/352994a7-d5cc-4932-8aee-f0ed1f173dbb-kube-api-access-rwz4b\") pod \"auth-test-model-9c75b6-kserve-5bdc976c55-vbzwg\" (UID: \"352994a7-d5cc-4932-8aee-f0ed1f173dbb\") " pod="ai-tenant-e2e-shared-b-9c75b6/auth-test-model-9c75b6-kserve-5bdc976c55-vbzwg"
Jul 12 17:43:56.505486 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:43:56.505458    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-9c75b6/auth-test-model-9c75b6-kserve-5bdc976c55-vbzwg"
Jul 12 17:43:56.632752 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:43:56.632725    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-9c75b6/auth-test-model-9c75b6-kserve-5bdc976c55-vbzwg"]
Jul 12 17:43:56.635233 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:43:56.635205    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod352994a7_d5cc_4932_8aee_f0ed1f173dbb.slice/crio-a9e5a08a6ded6d3bf8f7e610675bf414a8697b1e91e98275d863cbac96983e26 WatchSource:0}: Error finding container a9e5a08a6ded6d3bf8f7e610675bf414a8697b1e91e98275d863cbac96983e26: Status 404 returned error can't find the container with id a9e5a08a6ded6d3bf8f7e610675bf414a8697b1e91e98275d863cbac96983e26
Jul 12 17:43:57.531597 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:43:57.531560    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-9c75b6/auth-test-model-9c75b6-kserve-5bdc976c55-vbzwg" event={"ID":"352994a7-d5cc-4932-8aee-f0ed1f173dbb","Type":"ContainerStarted","Data":"9f7a83a3e6428d65fdfcadd4332f8c2d73135c118eadf31bae8b19136922538e"}
Jul 12 17:43:57.531597 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:43:57.531602    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-9c75b6/auth-test-model-9c75b6-kserve-5bdc976c55-vbzwg" event={"ID":"352994a7-d5cc-4932-8aee-f0ed1f173dbb","Type":"ContainerStarted","Data":"a9e5a08a6ded6d3bf8f7e610675bf414a8697b1e91e98275d863cbac96983e26"}
Jul 12 17:43:57.548870 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:43:57.548800    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-9c75b6/auth-test-model-9c75b6-kserve-5bdc976c55-vbzwg" podStartSLOduration=1.548780775 podStartE2EDuration="1.548780775s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 17:43:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 17:43:57.547020289 +0000 UTC m=+2445.667337085" watchObservedRunningTime="2026-07-12 17:43:57.548780775 +0000 UTC m=+2445.669097573"
Jul 12 17:43:59.883051 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:43:59.883015    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl4zn"]
Jul 12 17:44:06.506021 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:44:06.505991    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-9c75b6/auth-test-model-9c75b6-kserve-5bdc976c55-vbzwg"
Jul 12 17:44:06.506021 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:44:06.506030    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-9c75b6/auth-test-model-9c75b6-kserve-5bdc976c55-vbzwg"
Jul 12 17:44:06.518747 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:44:06.518719    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-9c75b6/auth-test-model-9c75b6-kserve-5bdc976c55-vbzwg"
Jul 12 17:44:06.585425 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:44:06.585395    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-9c75b6/auth-test-model-9c75b6-kserve-5bdc976c55-vbzwg"
Jul 12 17:44:13.588523 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:44:13.588476    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl4zn"]
Jul 12 17:44:30.284828 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:44:30.284793    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl4zn"]
Jul 12 17:44:50.094145 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:44:50.094107    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl4zn"]
Jul 12 17:45:07.287432 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:45:07.287351    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl4zn"]
Jul 12 17:45:22.388156 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:45:22.388118    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl4zn"]
Jul 12 17:45:41.087858 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:45:41.087823    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl4zn"]
Jul 12 17:45:53.573921 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:45:53.573889    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-e767e8/auth-test-model-e767e8-kserve-8847d9cc9-gzf7n"]
Jul 12 17:45:54.569814 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:45:54.569780    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-9c75b6/auth-test-model-9c75b6-kserve-5bdc976c55-vbzwg"]
Jul 12 17:45:57.290385 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:45:57.290352    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl4zn"]
Jul 12 17:46:08.593133 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:46:08.593066    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-e767e8/auth-test-model-e767e8-kserve-8847d9cc9-gzf7n" podUID="cc9cd027-7643-454f-a778-fdd4a29e5442" containerName="main" containerID="cri-o://072add77746c5b1844627013ec162d700a858787a8f8be1eabcbe2f2d9b621dd" gracePeriod=45
Jul 12 17:46:08.832708 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:46:08.832676    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-e767e8/auth-test-model-e767e8-kserve-8847d9cc9-gzf7n"
Jul 12 17:46:08.839403 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:46:08.839380    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc9cd027-7643-454f-a778-fdd4a29e5442-tmp-dir\") pod \"cc9cd027-7643-454f-a778-fdd4a29e5442\" (UID: \"cc9cd027-7643-454f-a778-fdd4a29e5442\") "
Jul 12 17:46:08.839534 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:46:08.839409    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bxxv4\" (UniqueName: \"kubernetes.io/projected/cc9cd027-7643-454f-a778-fdd4a29e5442-kube-api-access-bxxv4\") pod \"cc9cd027-7643-454f-a778-fdd4a29e5442\" (UID: \"cc9cd027-7643-454f-a778-fdd4a29e5442\") "
Jul 12 17:46:08.839534 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:46:08.839436    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc9cd027-7643-454f-a778-fdd4a29e5442-tls-certs\") pod \"cc9cd027-7643-454f-a778-fdd4a29e5442\" (UID: \"cc9cd027-7643-454f-a778-fdd4a29e5442\") "
Jul 12 17:46:08.839534 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:46:08.839451    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc9cd027-7643-454f-a778-fdd4a29e5442-dshm\") pod \"cc9cd027-7643-454f-a778-fdd4a29e5442\" (UID: \"cc9cd027-7643-454f-a778-fdd4a29e5442\") "
Jul 12 17:46:08.839534 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:46:08.839478    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc9cd027-7643-454f-a778-fdd4a29e5442-model-cache\") pod \"cc9cd027-7643-454f-a778-fdd4a29e5442\" (UID: \"cc9cd027-7643-454f-a778-fdd4a29e5442\") "
Jul 12 17:46:08.839763 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:46:08.839547    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc9cd027-7643-454f-a778-fdd4a29e5442-home\") pod \"cc9cd027-7643-454f-a778-fdd4a29e5442\" (UID: \"cc9cd027-7643-454f-a778-fdd4a29e5442\") "
Jul 12 17:46:08.839763 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:46:08.839676    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc9cd027-7643-454f-a778-fdd4a29e5442-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cc9cd027-7643-454f-a778-fdd4a29e5442" (UID: "cc9cd027-7643-454f-a778-fdd4a29e5442"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 12 17:46:08.839883 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:46:08.839808    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc9cd027-7643-454f-a778-fdd4a29e5442-model-cache" (OuterVolumeSpecName: "model-cache") pod "cc9cd027-7643-454f-a778-fdd4a29e5442" (UID: "cc9cd027-7643-454f-a778-fdd4a29e5442"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 12 17:46:08.839883 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:46:08.839846    2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc9cd027-7643-454f-a778-fdd4a29e5442-tmp-dir\") on node \"ip-10-0-141-94.ec2.internal\" DevicePath \"\""
Jul 12 17:46:08.839983 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:46:08.839873    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc9cd027-7643-454f-a778-fdd4a29e5442-home" (OuterVolumeSpecName: "home") pod "cc9cd027-7643-454f-a778-fdd4a29e5442" (UID: "cc9cd027-7643-454f-a778-fdd4a29e5442"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 12 17:46:08.841548 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:46:08.841527    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc9cd027-7643-454f-a778-fdd4a29e5442-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cc9cd027-7643-454f-a778-fdd4a29e5442" (UID: "cc9cd027-7643-454f-a778-fdd4a29e5442"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 12 17:46:08.841633 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:46:08.841599    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc9cd027-7643-454f-a778-fdd4a29e5442-kube-api-access-bxxv4" (OuterVolumeSpecName: "kube-api-access-bxxv4") pod "cc9cd027-7643-454f-a778-fdd4a29e5442" (UID: "cc9cd027-7643-454f-a778-fdd4a29e5442"). InnerVolumeSpecName "kube-api-access-bxxv4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 12 17:46:08.841633 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:46:08.841608    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc9cd027-7643-454f-a778-fdd4a29e5442-dshm" (OuterVolumeSpecName: "dshm") pod "cc9cd027-7643-454f-a778-fdd4a29e5442" (UID: "cc9cd027-7643-454f-a778-fdd4a29e5442"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 12 17:46:08.940688 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:46:08.940659    2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc9cd027-7643-454f-a778-fdd4a29e5442-home\") on node \"ip-10-0-141-94.ec2.internal\" DevicePath \"\""
Jul 12 17:46:08.940688 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:46:08.940681    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxxv4\" (UniqueName: \"kubernetes.io/projected/cc9cd027-7643-454f-a778-fdd4a29e5442-kube-api-access-bxxv4\") on node \"ip-10-0-141-94.ec2.internal\" DevicePath \"\""
Jul 12 17:46:08.940688 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:46:08.940692    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc9cd027-7643-454f-a778-fdd4a29e5442-tls-certs\") on node \"ip-10-0-141-94.ec2.internal\" DevicePath \"\""
Jul 12 17:46:08.940874 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:46:08.940701    2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc9cd027-7643-454f-a778-fdd4a29e5442-dshm\") on node \"ip-10-0-141-94.ec2.internal\" DevicePath \"\""
Jul 12 17:46:08.940874 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:46:08.940708    2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc9cd027-7643-454f-a778-fdd4a29e5442-model-cache\") on node \"ip-10-0-141-94.ec2.internal\" DevicePath \"\""
Jul 12 17:46:09.100682 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:46:09.100595    2569 generic.go:356] "Generic (PLEG): container finished" podID="cc9cd027-7643-454f-a778-fdd4a29e5442" containerID="072add77746c5b1844627013ec162d700a858787a8f8be1eabcbe2f2d9b621dd" exitCode=0
Jul 12 17:46:09.100682 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:46:09.100633    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-e767e8/auth-test-model-e767e8-kserve-8847d9cc9-gzf7n"
Jul 12 17:46:09.100908 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:46:09.100688    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-e767e8/auth-test-model-e767e8-kserve-8847d9cc9-gzf7n" event={"ID":"cc9cd027-7643-454f-a778-fdd4a29e5442","Type":"ContainerDied","Data":"072add77746c5b1844627013ec162d700a858787a8f8be1eabcbe2f2d9b621dd"}
Jul 12 17:46:09.100908 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:46:09.100736    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-e767e8/auth-test-model-e767e8-kserve-8847d9cc9-gzf7n" event={"ID":"cc9cd027-7643-454f-a778-fdd4a29e5442","Type":"ContainerDied","Data":"10439010b0cf283bfc48b18a9b0623029c38e15490dbd73dbdce1a5013e3b500"}
Jul 12 17:46:09.100908 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:46:09.100760    2569 scope.go:117] "RemoveContainer" containerID="072add77746c5b1844627013ec162d700a858787a8f8be1eabcbe2f2d9b621dd"
Jul 12 17:46:09.109435 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:46:09.109421    2569 scope.go:117] "RemoveContainer" containerID="072add77746c5b1844627013ec162d700a858787a8f8be1eabcbe2f2d9b621dd"
Jul 12 17:46:09.109687 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:46:09.109671    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"072add77746c5b1844627013ec162d700a858787a8f8be1eabcbe2f2d9b621dd\": container with ID starting with 072add77746c5b1844627013ec162d700a858787a8f8be1eabcbe2f2d9b621dd not found: ID does not exist" containerID="072add77746c5b1844627013ec162d700a858787a8f8be1eabcbe2f2d9b621dd"
Jul 12 17:46:09.109742 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:46:09.109693    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"072add77746c5b1844627013ec162d700a858787a8f8be1eabcbe2f2d9b621dd"} err="failed to get container status \"072add77746c5b1844627013ec162d700a858787a8f8be1eabcbe2f2d9b621dd\": rpc error: code = NotFound desc = could not find container \"072add77746c5b1844627013ec162d700a858787a8f8be1eabcbe2f2d9b621dd\": container with ID starting with 072add77746c5b1844627013ec162d700a858787a8f8be1eabcbe2f2d9b621dd not found: ID does not exist"
Jul 12 17:46:09.120165 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:46:09.120141    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-e767e8/auth-test-model-e767e8-kserve-8847d9cc9-gzf7n"]
Jul 12 17:46:09.125789 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:46:09.125771    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-e767e8/auth-test-model-e767e8-kserve-8847d9cc9-gzf7n"]
Jul 12 17:46:09.590395 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:46:09.590333    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-9c75b6/auth-test-model-9c75b6-kserve-5bdc976c55-vbzwg" podUID="352994a7-d5cc-4932-8aee-f0ed1f173dbb" containerName="main" containerID="cri-o://9f7a83a3e6428d65fdfcadd4332f8c2d73135c118eadf31bae8b19136922538e" gracePeriod=45
Jul 12 17:46:09.824305 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:46:09.824281    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-9c75b6/auth-test-model-9c75b6-kserve-5bdc976c55-vbzwg"
Jul 12 17:46:09.849486 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:46:09.849428    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/352994a7-d5cc-4932-8aee-f0ed1f173dbb-model-cache\") pod \"352994a7-d5cc-4932-8aee-f0ed1f173dbb\" (UID: \"352994a7-d5cc-4932-8aee-f0ed1f173dbb\") "
Jul 12 17:46:09.849604 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:46:09.849500    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/352994a7-d5cc-4932-8aee-f0ed1f173dbb-tls-certs\") pod \"352994a7-d5cc-4932-8aee-f0ed1f173dbb\" (UID: \"352994a7-d5cc-4932-8aee-f0ed1f173dbb\") "
Jul 12 17:46:09.849604 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:46:09.849542    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/352994a7-d5cc-4932-8aee-f0ed1f173dbb-home\") pod \"352994a7-d5cc-4932-8aee-f0ed1f173dbb\" (UID: \"352994a7-d5cc-4932-8aee-f0ed1f173dbb\") "
Jul 12 17:46:09.849604 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:46:09.849569    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rwz4b\" (UniqueName: \"kubernetes.io/projected/352994a7-d5cc-4932-8aee-f0ed1f173dbb-kube-api-access-rwz4b\") pod \"352994a7-d5cc-4932-8aee-f0ed1f173dbb\" (UID: \"352994a7-d5cc-4932-8aee-f0ed1f173dbb\") "
Jul 12 17:46:09.849604 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:46:09.849589    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/352994a7-d5cc-4932-8aee-f0ed1f173dbb-dshm\") pod \"352994a7-d5cc-4932-8aee-f0ed1f173dbb\" (UID: \"352994a7-d5cc-4932-8aee-f0ed1f173dbb\") "
Jul 12 17:46:09.849808 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:46:09.849622    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/352994a7-d5cc-4932-8aee-f0ed1f173dbb-tmp-dir\") pod \"352994a7-d5cc-4932-8aee-f0ed1f173dbb\" (UID: \"352994a7-d5cc-4932-8aee-f0ed1f173dbb\") "
Jul 12 17:46:09.849808 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:46:09.849645    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/352994a7-d5cc-4932-8aee-f0ed1f173dbb-model-cache" (OuterVolumeSpecName: "model-cache") pod "352994a7-d5cc-4932-8aee-f0ed1f173dbb" (UID: "352994a7-d5cc-4932-8aee-f0ed1f173dbb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 12 17:46:09.849911 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:46:09.849810    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/352994a7-d5cc-4932-8aee-f0ed1f173dbb-home" (OuterVolumeSpecName: "home") pod "352994a7-d5cc-4932-8aee-f0ed1f173dbb" (UID: "352994a7-d5cc-4932-8aee-f0ed1f173dbb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 12 17:46:09.849911 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:46:09.849855    2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/352994a7-d5cc-4932-8aee-f0ed1f173dbb-model-cache\") on node \"ip-10-0-141-94.ec2.internal\" DevicePath \"\""
Jul 12 17:46:09.850015 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:46:09.849978    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/352994a7-d5cc-4932-8aee-f0ed1f173dbb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "352994a7-d5cc-4932-8aee-f0ed1f173dbb" (UID: "352994a7-d5cc-4932-8aee-f0ed1f173dbb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 12 17:46:09.851644 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:46:09.851613    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/352994a7-d5cc-4932-8aee-f0ed1f173dbb-kube-api-access-rwz4b" (OuterVolumeSpecName: "kube-api-access-rwz4b") pod "352994a7-d5cc-4932-8aee-f0ed1f173dbb" (UID: "352994a7-d5cc-4932-8aee-f0ed1f173dbb"). InnerVolumeSpecName "kube-api-access-rwz4b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 12 17:46:09.851644 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:46:09.851618    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/352994a7-d5cc-4932-8aee-f0ed1f173dbb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "352994a7-d5cc-4932-8aee-f0ed1f173dbb" (UID: "352994a7-d5cc-4932-8aee-f0ed1f173dbb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 12 17:46:09.851880 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:46:09.851863    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/352994a7-d5cc-4932-8aee-f0ed1f173dbb-dshm" (OuterVolumeSpecName: "dshm") pod "352994a7-d5cc-4932-8aee-f0ed1f173dbb" (UID: "352994a7-d5cc-4932-8aee-f0ed1f173dbb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 12 17:46:09.950855 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:46:09.950831    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/352994a7-d5cc-4932-8aee-f0ed1f173dbb-tls-certs\") on node \"ip-10-0-141-94.ec2.internal\" DevicePath \"\""
Jul 12 17:46:09.950855 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:46:09.950851    2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/352994a7-d5cc-4932-8aee-f0ed1f173dbb-home\") on node \"ip-10-0-141-94.ec2.internal\" DevicePath \"\""
Jul 12 17:46:09.950992 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:46:09.950861    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwz4b\" (UniqueName: \"kubernetes.io/projected/352994a7-d5cc-4932-8aee-f0ed1f173dbb-kube-api-access-rwz4b\") on node \"ip-10-0-141-94.ec2.internal\" DevicePath \"\""
Jul 12 17:46:09.950992 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:46:09.950869    2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/352994a7-d5cc-4932-8aee-f0ed1f173dbb-dshm\") on node \"ip-10-0-141-94.ec2.internal\" DevicePath \"\""
Jul 12 17:46:09.950992 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:46:09.950877    2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/352994a7-d5cc-4932-8aee-f0ed1f173dbb-tmp-dir\") on node \"ip-10-0-141-94.ec2.internal\" DevicePath \"\""
Jul 12 17:46:10.106109 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:46:10.106036    2569 generic.go:356] "Generic (PLEG): container finished" podID="352994a7-d5cc-4932-8aee-f0ed1f173dbb" containerID="9f7a83a3e6428d65fdfcadd4332f8c2d73135c118eadf31bae8b19136922538e" exitCode=0
Jul 12 17:46:10.106109 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:46:10.106086    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-9c75b6/auth-test-model-9c75b6-kserve-5bdc976c55-vbzwg"
Jul 12 17:46:10.106296 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:46:10.106122    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-9c75b6/auth-test-model-9c75b6-kserve-5bdc976c55-vbzwg" event={"ID":"352994a7-d5cc-4932-8aee-f0ed1f173dbb","Type":"ContainerDied","Data":"9f7a83a3e6428d65fdfcadd4332f8c2d73135c118eadf31bae8b19136922538e"}
Jul 12 17:46:10.106296 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:46:10.106165    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-9c75b6/auth-test-model-9c75b6-kserve-5bdc976c55-vbzwg" event={"ID":"352994a7-d5cc-4932-8aee-f0ed1f173dbb","Type":"ContainerDied","Data":"a9e5a08a6ded6d3bf8f7e610675bf414a8697b1e91e98275d863cbac96983e26"}
Jul 12 17:46:10.106296 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:46:10.106181    2569 scope.go:117] "RemoveContainer" containerID="9f7a83a3e6428d65fdfcadd4332f8c2d73135c118eadf31bae8b19136922538e"
Jul 12 17:46:10.114702 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:46:10.114687    2569 scope.go:117] "RemoveContainer" containerID="9f7a83a3e6428d65fdfcadd4332f8c2d73135c118eadf31bae8b19136922538e"
Jul 12 17:46:10.114942 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:46:10.114928    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f7a83a3e6428d65fdfcadd4332f8c2d73135c118eadf31bae8b19136922538e\": container with ID starting with 9f7a83a3e6428d65fdfcadd4332f8c2d73135c118eadf31bae8b19136922538e not found: ID does not exist" containerID="9f7a83a3e6428d65fdfcadd4332f8c2d73135c118eadf31bae8b19136922538e"
Jul 12 17:46:10.115004 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:46:10.114949    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f7a83a3e6428d65fdfcadd4332f8c2d73135c118eadf31bae8b19136922538e"} err="failed to get container status \"9f7a83a3e6428d65fdfcadd4332f8c2d73135c118eadf31bae8b19136922538e\": rpc error: code = NotFound desc = could not find container \"9f7a83a3e6428d65fdfcadd4332f8c2d73135c118eadf31bae8b19136922538e\": container with ID starting with 9f7a83a3e6428d65fdfcadd4332f8c2d73135c118eadf31bae8b19136922538e not found: ID does not exist"
Jul 12 17:46:10.125577 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:46:10.125557    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-9c75b6/auth-test-model-9c75b6-kserve-5bdc976c55-vbzwg"]
Jul 12 17:46:10.128780 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:46:10.128757    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-9c75b6/auth-test-model-9c75b6-kserve-5bdc976c55-vbzwg"]
Jul 12 17:46:10.379557 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:46:10.379469    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="352994a7-d5cc-4932-8aee-f0ed1f173dbb" path="/var/lib/kubelet/pods/352994a7-d5cc-4932-8aee-f0ed1f173dbb/volumes"
Jul 12 17:46:10.379876 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:46:10.379861    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc9cd027-7643-454f-a778-fdd4a29e5442" path="/var/lib/kubelet/pods/cc9cd027-7643-454f-a778-fdd4a29e5442/volumes"
Jul 12 17:46:20.177905 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:46:20.177873    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-e767e8/sub-test-model-e767e8-kserve-f76fc4cbc-zb7b9"]
Jul 12 17:46:20.178275 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:46:20.178162    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc9cd027-7643-454f-a778-fdd4a29e5442" containerName="main"
Jul 12 17:46:20.178275 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:46:20.178172    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc9cd027-7643-454f-a778-fdd4a29e5442" containerName="main"
Jul 12 17:46:20.178275 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:46:20.178195    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="352994a7-d5cc-4932-8aee-f0ed1f173dbb" containerName="main"
Jul 12 17:46:20.178275 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:46:20.178200    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="352994a7-d5cc-4932-8aee-f0ed1f173dbb" containerName="main"
Jul 12 17:46:20.178275 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:46:20.178247    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="352994a7-d5cc-4932-8aee-f0ed1f173dbb" containerName="main"
Jul 12 17:46:20.178275 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:46:20.178256    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc9cd027-7643-454f-a778-fdd4a29e5442" containerName="main"
Jul 12 17:46:20.181033 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:46:20.181017    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-e767e8/sub-test-model-e767e8-kserve-f76fc4cbc-zb7b9"
Jul 12 17:46:20.184131 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:46:20.184105    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-e767e8\"/\"sub-test-model-e767e8-kserve-self-signed-certs\""
Jul 12 17:46:20.184251 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:46:20.184133    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-e767e8\"/\"kube-root-ca.crt\""
Jul 12 17:46:20.184251 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:46:20.184105    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-e767e8\"/\"openshift-service-ca.crt\""
Jul 12 17:46:20.184251 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:46:20.184107    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-e767e8\"/\"default-dockercfg-qd8cl\""
Jul 12 17:46:20.189487 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:46:20.189465    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-e767e8/sub-test-model-e767e8-kserve-f76fc4cbc-zb7b9"]
Jul 12 17:46:20.229296 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:46:20.229270    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45af3f34-9698-4dcf-80f1-11131a96037e-tls-certs\") pod \"sub-test-model-e767e8-kserve-f76fc4cbc-zb7b9\" (UID: \"45af3f34-9698-4dcf-80f1-11131a96037e\") " pod="ai-tenant-e2e-shared-a-e767e8/sub-test-model-e767e8-kserve-f76fc4cbc-zb7b9"
Jul 12 17:46:20.229390 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:46:20.229306    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45af3f34-9698-4dcf-80f1-11131a96037e-dshm\") pod \"sub-test-model-e767e8-kserve-f76fc4cbc-zb7b9\" (UID: \"45af3f34-9698-4dcf-80f1-11131a96037e\") " pod="ai-tenant-e2e-shared-a-e767e8/sub-test-model-e767e8-kserve-f76fc4cbc-zb7b9"
Jul 12 17:46:20.229390 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:46:20.229324    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfh6m\" (UniqueName: \"kubernetes.io/projected/45af3f34-9698-4dcf-80f1-11131a96037e-kube-api-access-nfh6m\") pod \"sub-test-model-e767e8-kserve-f76fc4cbc-zb7b9\" (UID: \"45af3f34-9698-4dcf-80f1-11131a96037e\") " pod="ai-tenant-e2e-shared-a-e767e8/sub-test-model-e767e8-kserve-f76fc4cbc-zb7b9"
Jul 12 17:46:20.229470 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:46:20.229405    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45af3f34-9698-4dcf-80f1-11131a96037e-model-cache\") pod \"sub-test-model-e767e8-kserve-f76fc4cbc-zb7b9\" (UID: \"45af3f34-9698-4dcf-80f1-11131a96037e\") " pod="ai-tenant-e2e-shared-a-e767e8/sub-test-model-e767e8-kserve-f76fc4cbc-zb7b9"
Jul 12 17:46:20.229470 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:46:20.229439    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45af3f34-9698-4dcf-80f1-11131a96037e-home\") pod \"sub-test-model-e767e8-kserve-f76fc4cbc-zb7b9\" (UID: \"45af3f34-9698-4dcf-80f1-11131a96037e\") " pod="ai-tenant-e2e-shared-a-e767e8/sub-test-model-e767e8-kserve-f76fc4cbc-zb7b9"
Jul 12 17:46:20.229470 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:46:20.229453    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45af3f34-9698-4dcf-80f1-11131a96037e-tmp-dir\") pod \"sub-test-model-e767e8-kserve-f76fc4cbc-zb7b9\" (UID: \"45af3f34-9698-4dcf-80f1-11131a96037e\") " pod="ai-tenant-e2e-shared-a-e767e8/sub-test-model-e767e8-kserve-f76fc4cbc-zb7b9"
Jul 12 17:46:20.330800 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:46:20.330771    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45af3f34-9698-4dcf-80f1-11131a96037e-home\") pod \"sub-test-model-e767e8-kserve-f76fc4cbc-zb7b9\" (UID: \"45af3f34-9698-4dcf-80f1-11131a96037e\") " pod="ai-tenant-e2e-shared-a-e767e8/sub-test-model-e767e8-kserve-f76fc4cbc-zb7b9"
Jul 12 17:46:20.330800 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:46:20.330802    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45af3f34-9698-4dcf-80f1-11131a96037e-tmp-dir\") pod \"sub-test-model-e767e8-kserve-f76fc4cbc-zb7b9\" (UID: \"45af3f34-9698-4dcf-80f1-11131a96037e\") " pod="ai-tenant-e2e-shared-a-e767e8/sub-test-model-e767e8-kserve-f76fc4cbc-zb7b9"
Jul 12 17:46:20.330980 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:46:20.330836    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45af3f34-9698-4dcf-80f1-11131a96037e-tls-certs\") pod \"sub-test-model-e767e8-kserve-f76fc4cbc-zb7b9\" (UID: \"45af3f34-9698-4dcf-80f1-11131a96037e\") " pod="ai-tenant-e2e-shared-a-e767e8/sub-test-model-e767e8-kserve-f76fc4cbc-zb7b9"
Jul 12 17:46:20.330980 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:46:20.330946    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45af3f34-9698-4dcf-80f1-11131a96037e-dshm\") pod \"sub-test-model-e767e8-kserve-f76fc4cbc-zb7b9\" (UID: \"45af3f34-9698-4dcf-80f1-11131a96037e\") " pod="ai-tenant-e2e-shared-a-e767e8/sub-test-model-e767e8-kserve-f76fc4cbc-zb7b9"
Jul 12 17:46:20.331063 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:46:20.330988    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfh6m\" (UniqueName: \"kubernetes.io/projected/45af3f34-9698-4dcf-80f1-11131a96037e-kube-api-access-nfh6m\") pod \"sub-test-model-e767e8-kserve-f76fc4cbc-zb7b9\" (UID: \"45af3f34-9698-4dcf-80f1-11131a96037e\") " pod="ai-tenant-e2e-shared-a-e767e8/sub-test-model-e767e8-kserve-f76fc4cbc-zb7b9"
Jul 12 17:46:20.331127 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:46:20.331096    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45af3f34-9698-4dcf-80f1-11131a96037e-model-cache\") pod \"sub-test-model-e767e8-kserve-f76fc4cbc-zb7b9\" (UID: \"45af3f34-9698-4dcf-80f1-11131a96037e\") " pod="ai-tenant-e2e-shared-a-e767e8/sub-test-model-e767e8-kserve-f76fc4cbc-zb7b9"
Jul 12 17:46:20.331239 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:46:20.331221    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45af3f34-9698-4dcf-80f1-11131a96037e-tmp-dir\") pod \"sub-test-model-e767e8-kserve-f76fc4cbc-zb7b9\" (UID: \"45af3f34-9698-4dcf-80f1-11131a96037e\") " pod="ai-tenant-e2e-shared-a-e767e8/sub-test-model-e767e8-kserve-f76fc4cbc-zb7b9"
Jul 12 17:46:20.331301 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:46:20.331263    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45af3f34-9698-4dcf-80f1-11131a96037e-home\") pod \"sub-test-model-e767e8-kserve-f76fc4cbc-zb7b9\" (UID: \"45af3f34-9698-4dcf-80f1-11131a96037e\") " pod="ai-tenant-e2e-shared-a-e767e8/sub-test-model-e767e8-kserve-f76fc4cbc-zb7b9"
Jul 12 17:46:20.331388 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:46:20.331374    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45af3f34-9698-4dcf-80f1-11131a96037e-model-cache\") pod \"sub-test-model-e767e8-kserve-f76fc4cbc-zb7b9\" (UID: \"45af3f34-9698-4dcf-80f1-11131a96037e\") " pod="ai-tenant-e2e-shared-a-e767e8/sub-test-model-e767e8-kserve-f76fc4cbc-zb7b9"
Jul 12 17:46:20.333132 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:46:20.333103    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45af3f34-9698-4dcf-80f1-11131a96037e-dshm\") pod \"sub-test-model-e767e8-kserve-f76fc4cbc-zb7b9\" (UID: \"45af3f34-9698-4dcf-80f1-11131a96037e\") " pod="ai-tenant-e2e-shared-a-e767e8/sub-test-model-e767e8-kserve-f76fc4cbc-zb7b9"
Jul 12 17:46:20.333449 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:46:20.333428    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45af3f34-9698-4dcf-80f1-11131a96037e-tls-certs\") pod \"sub-test-model-e767e8-kserve-f76fc4cbc-zb7b9\" (UID: \"45af3f34-9698-4dcf-80f1-11131a96037e\") " pod="ai-tenant-e2e-shared-a-e767e8/sub-test-model-e767e8-kserve-f76fc4cbc-zb7b9"
Jul 12 17:46:20.339599 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:46:20.339568    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfh6m\" (UniqueName: \"kubernetes.io/projected/45af3f34-9698-4dcf-80f1-11131a96037e-kube-api-access-nfh6m\") pod \"sub-test-model-e767e8-kserve-f76fc4cbc-zb7b9\" (UID: \"45af3f34-9698-4dcf-80f1-11131a96037e\") " pod="ai-tenant-e2e-shared-a-e767e8/sub-test-model-e767e8-kserve-f76fc4cbc-zb7b9"
Jul 12 17:46:20.491068 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:46:20.490989    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-e767e8/sub-test-model-e767e8-kserve-f76fc4cbc-zb7b9"
Jul 12 17:46:20.617541 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:46:20.617493    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-e767e8/sub-test-model-e767e8-kserve-f76fc4cbc-zb7b9"]
Jul 12 17:46:20.620129 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:46:20.620097    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod45af3f34_9698_4dcf_80f1_11131a96037e.slice/crio-0d106eaa65fe4aaf1474ece1c5c324d5f8e33076f56844b49aa5d62403cbc5c1 WatchSource:0}: Error finding container 0d106eaa65fe4aaf1474ece1c5c324d5f8e33076f56844b49aa5d62403cbc5c1: Status 404 returned error can't find the container with id 0d106eaa65fe4aaf1474ece1c5c324d5f8e33076f56844b49aa5d62403cbc5c1
Jul 12 17:46:21.156689 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:46:21.156655    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-e767e8/sub-test-model-e767e8-kserve-f76fc4cbc-zb7b9" event={"ID":"45af3f34-9698-4dcf-80f1-11131a96037e","Type":"ContainerStarted","Data":"e0c2053302040ec312f521a4bd37cbcc1db61ac8940787307410960770ff9be6"}
Jul 12 17:46:21.156842 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:46:21.156695    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-e767e8/sub-test-model-e767e8-kserve-f76fc4cbc-zb7b9" event={"ID":"45af3f34-9698-4dcf-80f1-11131a96037e","Type":"ContainerStarted","Data":"0d106eaa65fe4aaf1474ece1c5c324d5f8e33076f56844b49aa5d62403cbc5c1"}
Jul 12 17:46:21.171840 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:46:21.171802    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-e767e8/sub-test-model-e767e8-kserve-f76fc4cbc-zb7b9" podStartSLOduration=1.17178797 podStartE2EDuration="1.17178797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 17:46:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 17:46:21.17110149 +0000 UTC m=+2589.291418286" watchObservedRunningTime="2026-07-12 17:46:21.17178797 +0000 UTC m=+2589.292104766"
Jul 12 17:46:29.490333 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:46:29.490297    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl4zn"]
Jul 12 17:46:30.492097 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:46:30.492063    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-e767e8/sub-test-model-e767e8-kserve-f76fc4cbc-zb7b9"
Jul 12 17:46:30.492500 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:46:30.492272    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-e767e8/sub-test-model-e767e8-kserve-f76fc4cbc-zb7b9"
Jul 12 17:46:30.504489 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:46:30.504469    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-e767e8/sub-test-model-e767e8-kserve-f76fc4cbc-zb7b9"
Jul 12 17:46:31.208328 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:46:31.208294    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-e767e8/sub-test-model-e767e8-kserve-f76fc4cbc-zb7b9"
Jul 12 17:46:58.992376 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:46:58.992290    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-9c75b6/sub-test-model-9c75b6-kserve-69b5d55799-c4klq"]
Jul 12 17:46:58.995740 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:46:58.995721    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-9c75b6/sub-test-model-9c75b6-kserve-69b5d55799-c4klq"
Jul 12 17:46:58.998856 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:46:58.998830    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-9c75b6\"/\"openshift-service-ca.crt\""
Jul 12 17:46:58.998981 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:46:58.998839    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-9c75b6\"/\"kube-root-ca.crt\""
Jul 12 17:46:58.998981 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:46:58.998926    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-9c75b6\"/\"sub-test-model-9c75b6-kserve-self-signed-certs\""
Jul 12 17:46:58.998981 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:46:58.998929    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-9c75b6\"/\"default-dockercfg-n24tv\""
Jul 12 17:46:59.004269 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:46:59.004248    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-9c75b6/sub-test-model-9c75b6-kserve-69b5d55799-c4klq"]
Jul 12 17:46:59.156935 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:46:59.156900    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9bd98bbd-20e8-4370-878f-1d7bc2f0222d-home\") pod \"sub-test-model-9c75b6-kserve-69b5d55799-c4klq\" (UID: \"9bd98bbd-20e8-4370-878f-1d7bc2f0222d\") " pod="ai-tenant-e2e-shared-b-9c75b6/sub-test-model-9c75b6-kserve-69b5d55799-c4klq"
Jul 12 17:46:59.157102 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:46:59.156940    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9bd98bbd-20e8-4370-878f-1d7bc2f0222d-tmp-dir\") pod \"sub-test-model-9c75b6-kserve-69b5d55799-c4klq\" (UID: \"9bd98bbd-20e8-4370-878f-1d7bc2f0222d\") " pod="ai-tenant-e2e-shared-b-9c75b6/sub-test-model-9c75b6-kserve-69b5d55799-c4klq"
Jul 12 17:46:59.157102 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:46:59.156959    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9bd98bbd-20e8-4370-878f-1d7bc2f0222d-tls-certs\") pod \"sub-test-model-9c75b6-kserve-69b5d55799-c4klq\" (UID: \"9bd98bbd-20e8-4370-878f-1d7bc2f0222d\") " pod="ai-tenant-e2e-shared-b-9c75b6/sub-test-model-9c75b6-kserve-69b5d55799-c4klq"
Jul 12 17:46:59.157102 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:46:59.156977    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sm2dg\" (UniqueName: \"kubernetes.io/projected/9bd98bbd-20e8-4370-878f-1d7bc2f0222d-kube-api-access-sm2dg\") pod \"sub-test-model-9c75b6-kserve-69b5d55799-c4klq\" (UID: \"9bd98bbd-20e8-4370-878f-1d7bc2f0222d\") " pod="ai-tenant-e2e-shared-b-9c75b6/sub-test-model-9c75b6-kserve-69b5d55799-c4klq"
Jul 12 17:46:59.157102 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:46:59.157082    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9bd98bbd-20e8-4370-878f-1d7bc2f0222d-model-cache\") pod \"sub-test-model-9c75b6-kserve-69b5d55799-c4klq\" (UID: \"9bd98bbd-20e8-4370-878f-1d7bc2f0222d\") " pod="ai-tenant-e2e-shared-b-9c75b6/sub-test-model-9c75b6-kserve-69b5d55799-c4klq"
Jul 12 17:46:59.157257 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:46:59.157147    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9bd98bbd-20e8-4370-878f-1d7bc2f0222d-dshm\") pod \"sub-test-model-9c75b6-kserve-69b5d55799-c4klq\" (UID: \"9bd98bbd-20e8-4370-878f-1d7bc2f0222d\") " pod="ai-tenant-e2e-shared-b-9c75b6/sub-test-model-9c75b6-kserve-69b5d55799-c4klq"
Jul 12 17:46:59.257709 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:46:59.257636    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9bd98bbd-20e8-4370-878f-1d7bc2f0222d-model-cache\") pod \"sub-test-model-9c75b6-kserve-69b5d55799-c4klq\" (UID: \"9bd98bbd-20e8-4370-878f-1d7bc2f0222d\") " pod="ai-tenant-e2e-shared-b-9c75b6/sub-test-model-9c75b6-kserve-69b5d55799-c4klq"
Jul 12 17:46:59.257709 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:46:59.257688    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9bd98bbd-20e8-4370-878f-1d7bc2f0222d-dshm\") pod \"sub-test-model-9c75b6-kserve-69b5d55799-c4klq\" (UID: \"9bd98bbd-20e8-4370-878f-1d7bc2f0222d\") " pod="ai-tenant-e2e-shared-b-9c75b6/sub-test-model-9c75b6-kserve-69b5d55799-c4klq"
Jul 12 17:46:59.257709 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:46:59.257707    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9bd98bbd-20e8-4370-878f-1d7bc2f0222d-home\") pod \"sub-test-model-9c75b6-kserve-69b5d55799-c4klq\" (UID: \"9bd98bbd-20e8-4370-878f-1d7bc2f0222d\") " pod="ai-tenant-e2e-shared-b-9c75b6/sub-test-model-9c75b6-kserve-69b5d55799-c4klq"
Jul 12 17:46:59.257972 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:46:59.257727    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9bd98bbd-20e8-4370-878f-1d7bc2f0222d-tmp-dir\") pod \"sub-test-model-9c75b6-kserve-69b5d55799-c4klq\" (UID: \"9bd98bbd-20e8-4370-878f-1d7bc2f0222d\") " pod="ai-tenant-e2e-shared-b-9c75b6/sub-test-model-9c75b6-kserve-69b5d55799-c4klq"
Jul 12 17:46:59.257972 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:46:59.257743    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9bd98bbd-20e8-4370-878f-1d7bc2f0222d-tls-certs\") pod \"sub-test-model-9c75b6-kserve-69b5d55799-c4klq\" (UID: \"9bd98bbd-20e8-4370-878f-1d7bc2f0222d\") " pod="ai-tenant-e2e-shared-b-9c75b6/sub-test-model-9c75b6-kserve-69b5d55799-c4klq"
Jul 12 17:46:59.257972 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:46:59.257760    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sm2dg\" (UniqueName: \"kubernetes.io/projected/9bd98bbd-20e8-4370-878f-1d7bc2f0222d-kube-api-access-sm2dg\") pod \"sub-test-model-9c75b6-kserve-69b5d55799-c4klq\" (UID: \"9bd98bbd-20e8-4370-878f-1d7bc2f0222d\") " pod="ai-tenant-e2e-shared-b-9c75b6/sub-test-model-9c75b6-kserve-69b5d55799-c4klq"
Jul 12 17:46:59.258129 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:46:59.258061    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9bd98bbd-20e8-4370-878f-1d7bc2f0222d-model-cache\") pod \"sub-test-model-9c75b6-kserve-69b5d55799-c4klq\" (UID: \"9bd98bbd-20e8-4370-878f-1d7bc2f0222d\") " pod="ai-tenant-e2e-shared-b-9c75b6/sub-test-model-9c75b6-kserve-69b5d55799-c4klq"
Jul 12 17:46:59.258168 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:46:59.258149    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9bd98bbd-20e8-4370-878f-1d7bc2f0222d-tmp-dir\") pod \"sub-test-model-9c75b6-kserve-69b5d55799-c4klq\" (UID: \"9bd98bbd-20e8-4370-878f-1d7bc2f0222d\") " pod="ai-tenant-e2e-shared-b-9c75b6/sub-test-model-9c75b6-kserve-69b5d55799-c4klq"
Jul 12 17:46:59.258220 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:46:59.258199    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9bd98bbd-20e8-4370-878f-1d7bc2f0222d-home\") pod \"sub-test-model-9c75b6-kserve-69b5d55799-c4klq\" (UID: \"9bd98bbd-20e8-4370-878f-1d7bc2f0222d\") " pod="ai-tenant-e2e-shared-b-9c75b6/sub-test-model-9c75b6-kserve-69b5d55799-c4klq"
Jul 12 17:46:59.259936 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:46:59.259908    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9bd98bbd-20e8-4370-878f-1d7bc2f0222d-dshm\") pod \"sub-test-model-9c75b6-kserve-69b5d55799-c4klq\" (UID: \"9bd98bbd-20e8-4370-878f-1d7bc2f0222d\") " pod="ai-tenant-e2e-shared-b-9c75b6/sub-test-model-9c75b6-kserve-69b5d55799-c4klq"
Jul 12 17:46:59.260326 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:46:59.260302    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9bd98bbd-20e8-4370-878f-1d7bc2f0222d-tls-certs\") pod \"sub-test-model-9c75b6-kserve-69b5d55799-c4klq\" (UID: \"9bd98bbd-20e8-4370-878f-1d7bc2f0222d\") " pod="ai-tenant-e2e-shared-b-9c75b6/sub-test-model-9c75b6-kserve-69b5d55799-c4klq"
Jul 12 17:46:59.265767 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:46:59.265748    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sm2dg\" (UniqueName: \"kubernetes.io/projected/9bd98bbd-20e8-4370-878f-1d7bc2f0222d-kube-api-access-sm2dg\") pod \"sub-test-model-9c75b6-kserve-69b5d55799-c4klq\" (UID: \"9bd98bbd-20e8-4370-878f-1d7bc2f0222d\") " pod="ai-tenant-e2e-shared-b-9c75b6/sub-test-model-9c75b6-kserve-69b5d55799-c4klq"
Jul 12 17:46:59.306148 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:46:59.306123    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-9c75b6/sub-test-model-9c75b6-kserve-69b5d55799-c4klq"
Jul 12 17:46:59.432228 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:46:59.432206    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-9c75b6/sub-test-model-9c75b6-kserve-69b5d55799-c4klq"]
Jul 12 17:46:59.434393 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:46:59.434362    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9bd98bbd_20e8_4370_878f_1d7bc2f0222d.slice/crio-a64b9b3b00088acbcd177764f7694bcc164d9a9376a382f5816072bbdc45213e WatchSource:0}: Error finding container a64b9b3b00088acbcd177764f7694bcc164d9a9376a382f5816072bbdc45213e: Status 404 returned error can't find the container with id a64b9b3b00088acbcd177764f7694bcc164d9a9376a382f5816072bbdc45213e
Jul 12 17:47:00.323544 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:47:00.323488    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-9c75b6/sub-test-model-9c75b6-kserve-69b5d55799-c4klq" event={"ID":"9bd98bbd-20e8-4370-878f-1d7bc2f0222d","Type":"ContainerStarted","Data":"9630cb7080591e4be794833e587e317410f347a2cacf2543487e5ef9c625b274"}
Jul 12 17:47:00.323946 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:47:00.323566    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-9c75b6/sub-test-model-9c75b6-kserve-69b5d55799-c4klq" event={"ID":"9bd98bbd-20e8-4370-878f-1d7bc2f0222d","Type":"ContainerStarted","Data":"a64b9b3b00088acbcd177764f7694bcc164d9a9376a382f5816072bbdc45213e"}
Jul 12 17:47:00.339744 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:47:00.339691    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-9c75b6/sub-test-model-9c75b6-kserve-69b5d55799-c4klq" podStartSLOduration=2.339675075 podStartE2EDuration="2.339675075s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 17:46:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 17:47:00.338492961 +0000 UTC m=+2628.458809772" watchObservedRunningTime="2026-07-12 17:47:00.339675075 +0000 UTC m=+2628.459991901"
Jul 12 17:47:02.995853 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:47:02.995818    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl4zn"]
Jul 12 17:47:09.306680 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:47:09.306629    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-9c75b6/sub-test-model-9c75b6-kserve-69b5d55799-c4klq"
Jul 12 17:47:09.306680 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:47:09.306691    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-9c75b6/sub-test-model-9c75b6-kserve-69b5d55799-c4klq"
Jul 12 17:47:09.319169 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:47:09.319141    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-9c75b6/sub-test-model-9c75b6-kserve-69b5d55799-c4klq"
Jul 12 17:47:09.371611 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:47:09.371576    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-9c75b6/sub-test-model-9c75b6-kserve-69b5d55799-c4klq"
Jul 12 17:47:12.514001 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:47:12.513971    2569 scope.go:117] "RemoveContainer" containerID="3be359da825a11ac848ca6e725230af6259c45e8fb9b2ecf9927d6ef15b197c9"
Jul 12 17:47:28.384919 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:47:28.384882    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl4zn"]
Jul 12 17:47:48.087105 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:47:48.087065    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl4zn"]
Jul 12 17:47:52.716356 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:47:52.716322    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-e767e8/sub-test-model-e767e8-kserve-f76fc4cbc-zb7b9"]
Jul 12 17:47:53.784360 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:47:53.784326    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-9c75b6/sub-test-model-9c75b6-kserve-69b5d55799-c4klq"]
Jul 12 17:47:56.798826 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:47:56.798797    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-e767e8/rate-test-model-e767e8-kserve-6895b4f675-q2qmr"]
Jul 12 17:47:56.801979 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:47:56.801962    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-e767e8/rate-test-model-e767e8-kserve-6895b4f675-q2qmr"
Jul 12 17:47:56.804236 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:47:56.804216    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-e767e8\"/\"rate-test-model-e767e8-kserve-self-signed-certs\""
Jul 12 17:47:56.809946 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:47:56.809923    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-e767e8/rate-test-model-e767e8-kserve-6895b4f675-q2qmr"]
Jul 12 17:47:56.836348 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:47:56.836325    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/49e8743b-e8bb-4f9e-b498-7f700c469381-home\") pod \"rate-test-model-e767e8-kserve-6895b4f675-q2qmr\" (UID: \"49e8743b-e8bb-4f9e-b498-7f700c469381\") " pod="ai-tenant-e2e-shared-a-e767e8/rate-test-model-e767e8-kserve-6895b4f675-q2qmr"
Jul 12 17:47:56.836444 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:47:56.836353    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/49e8743b-e8bb-4f9e-b498-7f700c469381-model-cache\") pod \"rate-test-model-e767e8-kserve-6895b4f675-q2qmr\" (UID: \"49e8743b-e8bb-4f9e-b498-7f700c469381\") " pod="ai-tenant-e2e-shared-a-e767e8/rate-test-model-e767e8-kserve-6895b4f675-q2qmr"
Jul 12 17:47:56.836444 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:47:56.836400    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccrtg\" (UniqueName: \"kubernetes.io/projected/49e8743b-e8bb-4f9e-b498-7f700c469381-kube-api-access-ccrtg\") pod \"rate-test-model-e767e8-kserve-6895b4f675-q2qmr\" (UID: \"49e8743b-e8bb-4f9e-b498-7f700c469381\") " pod="ai-tenant-e2e-shared-a-e767e8/rate-test-model-e767e8-kserve-6895b4f675-q2qmr"
Jul 12 17:47:56.836569 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:47:56.836459    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/49e8743b-e8bb-4f9e-b498-7f700c469381-dshm\") pod \"rate-test-model-e767e8-kserve-6895b4f675-q2qmr\" (UID: \"49e8743b-e8bb-4f9e-b498-7f700c469381\") " pod="ai-tenant-e2e-shared-a-e767e8/rate-test-model-e767e8-kserve-6895b4f675-q2qmr"
Jul 12 17:47:56.836569 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:47:56.836485    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/49e8743b-e8bb-4f9e-b498-7f700c469381-tls-certs\") pod \"rate-test-model-e767e8-kserve-6895b4f675-q2qmr\" (UID: \"49e8743b-e8bb-4f9e-b498-7f700c469381\") " pod="ai-tenant-e2e-shared-a-e767e8/rate-test-model-e767e8-kserve-6895b4f675-q2qmr"
Jul 12 17:47:56.836569 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:47:56.836528    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49e8743b-e8bb-4f9e-b498-7f700c469381-tmp-dir\") pod \"rate-test-model-e767e8-kserve-6895b4f675-q2qmr\" (UID: \"49e8743b-e8bb-4f9e-b498-7f700c469381\") " pod="ai-tenant-e2e-shared-a-e767e8/rate-test-model-e767e8-kserve-6895b4f675-q2qmr"
Jul 12 17:47:56.937631 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:47:56.937606    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccrtg\" (UniqueName: \"kubernetes.io/projected/49e8743b-e8bb-4f9e-b498-7f700c469381-kube-api-access-ccrtg\") pod \"rate-test-model-e767e8-kserve-6895b4f675-q2qmr\" (UID: \"49e8743b-e8bb-4f9e-b498-7f700c469381\") " pod="ai-tenant-e2e-shared-a-e767e8/rate-test-model-e767e8-kserve-6895b4f675-q2qmr"
Jul 12 17:47:56.937762 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:47:56.937644    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/49e8743b-e8bb-4f9e-b498-7f700c469381-dshm\") pod \"rate-test-model-e767e8-kserve-6895b4f675-q2qmr\" (UID: \"49e8743b-e8bb-4f9e-b498-7f700c469381\") " pod="ai-tenant-e2e-shared-a-e767e8/rate-test-model-e767e8-kserve-6895b4f675-q2qmr"
Jul 12 17:47:56.937762 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:47:56.937661    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/49e8743b-e8bb-4f9e-b498-7f700c469381-tls-certs\") pod \"rate-test-model-e767e8-kserve-6895b4f675-q2qmr\" (UID: \"49e8743b-e8bb-4f9e-b498-7f700c469381\") " pod="ai-tenant-e2e-shared-a-e767e8/rate-test-model-e767e8-kserve-6895b4f675-q2qmr"
Jul 12 17:47:56.937762 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:47:56.937680    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49e8743b-e8bb-4f9e-b498-7f700c469381-tmp-dir\") pod \"rate-test-model-e767e8-kserve-6895b4f675-q2qmr\" (UID: \"49e8743b-e8bb-4f9e-b498-7f700c469381\") " pod="ai-tenant-e2e-shared-a-e767e8/rate-test-model-e767e8-kserve-6895b4f675-q2qmr"
Jul 12 17:47:56.937762 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:47:56.937730    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/49e8743b-e8bb-4f9e-b498-7f700c469381-home\") pod \"rate-test-model-e767e8-kserve-6895b4f675-q2qmr\" (UID: \"49e8743b-e8bb-4f9e-b498-7f700c469381\") " pod="ai-tenant-e2e-shared-a-e767e8/rate-test-model-e767e8-kserve-6895b4f675-q2qmr"
Jul 12 17:47:56.937762 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:47:56.937756    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/49e8743b-e8bb-4f9e-b498-7f700c469381-model-cache\") pod \"rate-test-model-e767e8-kserve-6895b4f675-q2qmr\" (UID: \"49e8743b-e8bb-4f9e-b498-7f700c469381\") " pod="ai-tenant-e2e-shared-a-e767e8/rate-test-model-e767e8-kserve-6895b4f675-q2qmr"
Jul 12 17:47:56.938127 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:47:56.938103    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/49e8743b-e8bb-4f9e-b498-7f700c469381-home\") pod \"rate-test-model-e767e8-kserve-6895b4f675-q2qmr\" (UID: \"49e8743b-e8bb-4f9e-b498-7f700c469381\") " pod="ai-tenant-e2e-shared-a-e767e8/rate-test-model-e767e8-kserve-6895b4f675-q2qmr"
Jul 12 17:47:56.938192 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:47:56.938123    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/49e8743b-e8bb-4f9e-b498-7f700c469381-model-cache\") pod \"rate-test-model-e767e8-kserve-6895b4f675-q2qmr\" (UID: \"49e8743b-e8bb-4f9e-b498-7f700c469381\") " pod="ai-tenant-e2e-shared-a-e767e8/rate-test-model-e767e8-kserve-6895b4f675-q2qmr"
Jul 12 17:47:56.938192 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:47:56.938181    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49e8743b-e8bb-4f9e-b498-7f700c469381-tmp-dir\") pod \"rate-test-model-e767e8-kserve-6895b4f675-q2qmr\" (UID: \"49e8743b-e8bb-4f9e-b498-7f700c469381\") " pod="ai-tenant-e2e-shared-a-e767e8/rate-test-model-e767e8-kserve-6895b4f675-q2qmr"
Jul 12 17:47:56.939873 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:47:56.939850    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/49e8743b-e8bb-4f9e-b498-7f700c469381-dshm\") pod \"rate-test-model-e767e8-kserve-6895b4f675-q2qmr\" (UID: \"49e8743b-e8bb-4f9e-b498-7f700c469381\") " pod="ai-tenant-e2e-shared-a-e767e8/rate-test-model-e767e8-kserve-6895b4f675-q2qmr"
Jul 12 17:47:56.940120 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:47:56.940102    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/49e8743b-e8bb-4f9e-b498-7f700c469381-tls-certs\") pod \"rate-test-model-e767e8-kserve-6895b4f675-q2qmr\" (UID: \"49e8743b-e8bb-4f9e-b498-7f700c469381\") " pod="ai-tenant-e2e-shared-a-e767e8/rate-test-model-e767e8-kserve-6895b4f675-q2qmr"
Jul 12 17:47:56.944761 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:47:56.944744    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccrtg\" (UniqueName: \"kubernetes.io/projected/49e8743b-e8bb-4f9e-b498-7f700c469381-kube-api-access-ccrtg\") pod \"rate-test-model-e767e8-kserve-6895b4f675-q2qmr\" (UID: \"49e8743b-e8bb-4f9e-b498-7f700c469381\") " pod="ai-tenant-e2e-shared-a-e767e8/rate-test-model-e767e8-kserve-6895b4f675-q2qmr"
Jul 12 17:47:57.111473 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:47:57.111412    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-e767e8/rate-test-model-e767e8-kserve-6895b4f675-q2qmr"
Jul 12 17:47:57.237910 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:47:57.237882    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-e767e8/rate-test-model-e767e8-kserve-6895b4f675-q2qmr"]
Jul 12 17:47:57.240346 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:47:57.240321    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod49e8743b_e8bb_4f9e_b498_7f700c469381.slice/crio-a0d83d87e4b35fe310217fac302310a7294a82ed8b505edccaaf92bdcd9f4900 WatchSource:0}: Error finding container a0d83d87e4b35fe310217fac302310a7294a82ed8b505edccaaf92bdcd9f4900: Status 404 returned error can't find the container with id a0d83d87e4b35fe310217fac302310a7294a82ed8b505edccaaf92bdcd9f4900
Jul 12 17:47:57.560204 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:47:57.560173    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-e767e8/rate-test-model-e767e8-kserve-6895b4f675-q2qmr" event={"ID":"49e8743b-e8bb-4f9e-b498-7f700c469381","Type":"ContainerStarted","Data":"2977ebfd3cef05a1a32025551589f9e09953d95fd25ea041dd86acfc86c00a59"}
Jul 12 17:47:57.560204 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:47:57.560211    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-e767e8/rate-test-model-e767e8-kserve-6895b4f675-q2qmr" event={"ID":"49e8743b-e8bb-4f9e-b498-7f700c469381","Type":"ContainerStarted","Data":"a0d83d87e4b35fe310217fac302310a7294a82ed8b505edccaaf92bdcd9f4900"}
Jul 12 17:47:57.575369 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:47:57.575324    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-e767e8/rate-test-model-e767e8-kserve-6895b4f675-q2qmr" podStartSLOduration=1.5753098140000001 podStartE2EDuration="1.575309814s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 17:47:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 17:47:57.574554274 +0000 UTC m=+2685.694871074" watchObservedRunningTime="2026-07-12 17:47:57.575309814 +0000 UTC m=+2685.695626611"
Jul 12 17:48:06.093104 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:48:06.093024    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl4zn"]
Jul 12 17:48:07.111636 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:48:07.111600    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-e767e8/rate-test-model-e767e8-kserve-6895b4f675-q2qmr"
Jul 12 17:48:07.112046 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:48:07.111652    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-e767e8/rate-test-model-e767e8-kserve-6895b4f675-q2qmr"
Jul 12 17:48:07.124197 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:48:07.124165    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-e767e8/rate-test-model-e767e8-kserve-6895b4f675-q2qmr"
Jul 12 17:48:07.613683 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:48:07.613653    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-e767e8/rate-test-model-e767e8-kserve-6895b4f675-q2qmr"
Jul 12 17:48:07.738087 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:48:07.738033    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-e767e8/sub-test-model-e767e8-kserve-f76fc4cbc-zb7b9" podUID="45af3f34-9698-4dcf-80f1-11131a96037e" containerName="main" containerID="cri-o://e0c2053302040ec312f521a4bd37cbcc1db61ac8940787307410960770ff9be6" gracePeriod=45
Jul 12 17:48:07.982462 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:48:07.982441    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-e767e8/sub-test-model-e767e8-kserve-f76fc4cbc-zb7b9"
Jul 12 17:48:08.023902 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:48:08.023873    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45af3f34-9698-4dcf-80f1-11131a96037e-tmp-dir\") pod \"45af3f34-9698-4dcf-80f1-11131a96037e\" (UID: \"45af3f34-9698-4dcf-80f1-11131a96037e\") "
Jul 12 17:48:08.024032 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:48:08.023928    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45af3f34-9698-4dcf-80f1-11131a96037e-dshm\") pod \"45af3f34-9698-4dcf-80f1-11131a96037e\" (UID: \"45af3f34-9698-4dcf-80f1-11131a96037e\") "
Jul 12 17:48:08.024032 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:48:08.023960    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nfh6m\" (UniqueName: \"kubernetes.io/projected/45af3f34-9698-4dcf-80f1-11131a96037e-kube-api-access-nfh6m\") pod \"45af3f34-9698-4dcf-80f1-11131a96037e\" (UID: \"45af3f34-9698-4dcf-80f1-11131a96037e\") "
Jul 12 17:48:08.024032 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:48:08.024005    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45af3f34-9698-4dcf-80f1-11131a96037e-model-cache\") pod \"45af3f34-9698-4dcf-80f1-11131a96037e\" (UID: \"45af3f34-9698-4dcf-80f1-11131a96037e\") "
Jul 12 17:48:08.024032 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:48:08.024030    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45af3f34-9698-4dcf-80f1-11131a96037e-home\") pod \"45af3f34-9698-4dcf-80f1-11131a96037e\" (UID: \"45af3f34-9698-4dcf-80f1-11131a96037e\") "
Jul 12 17:48:08.024254 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:48:08.024124    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45af3f34-9698-4dcf-80f1-11131a96037e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "45af3f34-9698-4dcf-80f1-11131a96037e" (UID: "45af3f34-9698-4dcf-80f1-11131a96037e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 12 17:48:08.024254 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:48:08.024157    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45af3f34-9698-4dcf-80f1-11131a96037e-tls-certs\") pod \"45af3f34-9698-4dcf-80f1-11131a96037e\" (UID: \"45af3f34-9698-4dcf-80f1-11131a96037e\") "
Jul 12 17:48:08.024341 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:48:08.024267    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45af3f34-9698-4dcf-80f1-11131a96037e-home" (OuterVolumeSpecName: "home") pod "45af3f34-9698-4dcf-80f1-11131a96037e" (UID: "45af3f34-9698-4dcf-80f1-11131a96037e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 12 17:48:08.024341 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:48:08.024285    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45af3f34-9698-4dcf-80f1-11131a96037e-model-cache" (OuterVolumeSpecName: "model-cache") pod "45af3f34-9698-4dcf-80f1-11131a96037e" (UID: "45af3f34-9698-4dcf-80f1-11131a96037e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 12 17:48:08.024457 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:48:08.024438    2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45af3f34-9698-4dcf-80f1-11131a96037e-model-cache\") on node \"ip-10-0-141-94.ec2.internal\" DevicePath \"\""
Jul 12 17:48:08.024457 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:48:08.024454    2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45af3f34-9698-4dcf-80f1-11131a96037e-home\") on node \"ip-10-0-141-94.ec2.internal\" DevicePath \"\""
Jul 12 17:48:08.024559 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:48:08.024466    2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45af3f34-9698-4dcf-80f1-11131a96037e-tmp-dir\") on node \"ip-10-0-141-94.ec2.internal\" DevicePath \"\""
Jul 12 17:48:08.026036 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:48:08.026009    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45af3f34-9698-4dcf-80f1-11131a96037e-kube-api-access-nfh6m" (OuterVolumeSpecName: "kube-api-access-nfh6m") pod "45af3f34-9698-4dcf-80f1-11131a96037e" (UID: "45af3f34-9698-4dcf-80f1-11131a96037e"). InnerVolumeSpecName "kube-api-access-nfh6m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 12 17:48:08.026430 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:48:08.026405    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45af3f34-9698-4dcf-80f1-11131a96037e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "45af3f34-9698-4dcf-80f1-11131a96037e" (UID: "45af3f34-9698-4dcf-80f1-11131a96037e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 12 17:48:08.026502 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:48:08.026411    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45af3f34-9698-4dcf-80f1-11131a96037e-dshm" (OuterVolumeSpecName: "dshm") pod "45af3f34-9698-4dcf-80f1-11131a96037e" (UID: "45af3f34-9698-4dcf-80f1-11131a96037e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 12 17:48:08.125218 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:48:08.125192    2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45af3f34-9698-4dcf-80f1-11131a96037e-dshm\") on node \"ip-10-0-141-94.ec2.internal\" DevicePath \"\""
Jul 12 17:48:08.125218 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:48:08.125216    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfh6m\" (UniqueName: \"kubernetes.io/projected/45af3f34-9698-4dcf-80f1-11131a96037e-kube-api-access-nfh6m\") on node \"ip-10-0-141-94.ec2.internal\" DevicePath \"\""
Jul 12 17:48:08.125581 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:48:08.125228    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45af3f34-9698-4dcf-80f1-11131a96037e-tls-certs\") on node \"ip-10-0-141-94.ec2.internal\" DevicePath \"\""
Jul 12 17:48:08.607686 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:48:08.607652    2569 generic.go:356] "Generic (PLEG): container finished" podID="45af3f34-9698-4dcf-80f1-11131a96037e" containerID="e0c2053302040ec312f521a4bd37cbcc1db61ac8940787307410960770ff9be6" exitCode=0
Jul 12 17:48:08.607840 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:48:08.607698    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-e767e8/sub-test-model-e767e8-kserve-f76fc4cbc-zb7b9"
Jul 12 17:48:08.607840 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:48:08.607736    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-e767e8/sub-test-model-e767e8-kserve-f76fc4cbc-zb7b9" event={"ID":"45af3f34-9698-4dcf-80f1-11131a96037e","Type":"ContainerDied","Data":"e0c2053302040ec312f521a4bd37cbcc1db61ac8940787307410960770ff9be6"}
Jul 12 17:48:08.607840 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:48:08.607791    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-e767e8/sub-test-model-e767e8-kserve-f76fc4cbc-zb7b9" event={"ID":"45af3f34-9698-4dcf-80f1-11131a96037e","Type":"ContainerDied","Data":"0d106eaa65fe4aaf1474ece1c5c324d5f8e33076f56844b49aa5d62403cbc5c1"}
Jul 12 17:48:08.607840 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:48:08.607819    2569 scope.go:117] "RemoveContainer" containerID="e0c2053302040ec312f521a4bd37cbcc1db61ac8940787307410960770ff9be6"
Jul 12 17:48:08.615685 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:48:08.615669    2569 scope.go:117] "RemoveContainer" containerID="e0c2053302040ec312f521a4bd37cbcc1db61ac8940787307410960770ff9be6"
Jul 12 17:48:08.615951 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:48:08.615932    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0c2053302040ec312f521a4bd37cbcc1db61ac8940787307410960770ff9be6\": container with ID starting with e0c2053302040ec312f521a4bd37cbcc1db61ac8940787307410960770ff9be6 not found: ID does not exist" containerID="e0c2053302040ec312f521a4bd37cbcc1db61ac8940787307410960770ff9be6"
Jul 12 17:48:08.616027 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:48:08.615961    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0c2053302040ec312f521a4bd37cbcc1db61ac8940787307410960770ff9be6"} err="failed to get container status \"e0c2053302040ec312f521a4bd37cbcc1db61ac8940787307410960770ff9be6\": rpc error: code = NotFound desc = could not find container \"e0c2053302040ec312f521a4bd37cbcc1db61ac8940787307410960770ff9be6\": container with ID starting with e0c2053302040ec312f521a4bd37cbcc1db61ac8940787307410960770ff9be6 not found: ID does not exist"
Jul 12 17:48:08.622672 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:48:08.622649    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-e767e8/sub-test-model-e767e8-kserve-f76fc4cbc-zb7b9"]
Jul 12 17:48:08.624425 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:48:08.624404    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-e767e8/sub-test-model-e767e8-kserve-f76fc4cbc-zb7b9"]
Jul 12 17:48:08.803909 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:48:08.803869    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-9c75b6/sub-test-model-9c75b6-kserve-69b5d55799-c4klq" podUID="9bd98bbd-20e8-4370-878f-1d7bc2f0222d" containerName="main" containerID="cri-o://9630cb7080591e4be794833e587e317410f347a2cacf2543487e5ef9c625b274" gracePeriod=45
Jul 12 17:48:09.049856 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:48:09.049829    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-9c75b6/sub-test-model-9c75b6-kserve-69b5d55799-c4klq"
Jul 12 17:48:09.134971 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:48:09.134913    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sm2dg\" (UniqueName: \"kubernetes.io/projected/9bd98bbd-20e8-4370-878f-1d7bc2f0222d-kube-api-access-sm2dg\") pod \"9bd98bbd-20e8-4370-878f-1d7bc2f0222d\" (UID: \"9bd98bbd-20e8-4370-878f-1d7bc2f0222d\") "
Jul 12 17:48:09.134971 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:48:09.134958    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9bd98bbd-20e8-4370-878f-1d7bc2f0222d-model-cache\") pod \"9bd98bbd-20e8-4370-878f-1d7bc2f0222d\" (UID: \"9bd98bbd-20e8-4370-878f-1d7bc2f0222d\") "
Jul 12 17:48:09.135318 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:48:09.134984    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9bd98bbd-20e8-4370-878f-1d7bc2f0222d-tls-certs\") pod \"9bd98bbd-20e8-4370-878f-1d7bc2f0222d\" (UID: \"9bd98bbd-20e8-4370-878f-1d7bc2f0222d\") "
Jul 12 17:48:09.135318 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:48:09.135106    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9bd98bbd-20e8-4370-878f-1d7bc2f0222d-home\") pod \"9bd98bbd-20e8-4370-878f-1d7bc2f0222d\" (UID: \"9bd98bbd-20e8-4370-878f-1d7bc2f0222d\") "
Jul 12 17:48:09.135318 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:48:09.135140    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9bd98bbd-20e8-4370-878f-1d7bc2f0222d-tmp-dir\") pod \"9bd98bbd-20e8-4370-878f-1d7bc2f0222d\" (UID: \"9bd98bbd-20e8-4370-878f-1d7bc2f0222d\") "
Jul 12 17:48:09.135318 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:48:09.135178    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bd98bbd-20e8-4370-878f-1d7bc2f0222d-model-cache" (OuterVolumeSpecName: "model-cache") pod "9bd98bbd-20e8-4370-878f-1d7bc2f0222d" (UID: "9bd98bbd-20e8-4370-878f-1d7bc2f0222d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 12 17:48:09.135318 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:48:09.135193    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9bd98bbd-20e8-4370-878f-1d7bc2f0222d-dshm\") pod \"9bd98bbd-20e8-4370-878f-1d7bc2f0222d\" (UID: \"9bd98bbd-20e8-4370-878f-1d7bc2f0222d\") "
Jul 12 17:48:09.135602 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:48:09.135331    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bd98bbd-20e8-4370-878f-1d7bc2f0222d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9bd98bbd-20e8-4370-878f-1d7bc2f0222d" (UID: "9bd98bbd-20e8-4370-878f-1d7bc2f0222d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 12 17:48:09.135602 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:48:09.135358    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bd98bbd-20e8-4370-878f-1d7bc2f0222d-home" (OuterVolumeSpecName: "home") pod "9bd98bbd-20e8-4370-878f-1d7bc2f0222d" (UID: "9bd98bbd-20e8-4370-878f-1d7bc2f0222d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 12 17:48:09.135602 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:48:09.135530    2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9bd98bbd-20e8-4370-878f-1d7bc2f0222d-model-cache\") on node \"ip-10-0-141-94.ec2.internal\" DevicePath \"\""
Jul 12 17:48:09.135602 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:48:09.135543    2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9bd98bbd-20e8-4370-878f-1d7bc2f0222d-home\") on node \"ip-10-0-141-94.ec2.internal\" DevicePath \"\""
Jul 12 17:48:09.135602 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:48:09.135551    2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9bd98bbd-20e8-4370-878f-1d7bc2f0222d-tmp-dir\") on node \"ip-10-0-141-94.ec2.internal\" DevicePath \"\""
Jul 12 17:48:09.137073 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:48:09.137048    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9bd98bbd-20e8-4370-878f-1d7bc2f0222d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9bd98bbd-20e8-4370-878f-1d7bc2f0222d" (UID: "9bd98bbd-20e8-4370-878f-1d7bc2f0222d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 12 17:48:09.137168 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:48:09.137092    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bd98bbd-20e8-4370-878f-1d7bc2f0222d-dshm" (OuterVolumeSpecName: "dshm") pod "9bd98bbd-20e8-4370-878f-1d7bc2f0222d" (UID: "9bd98bbd-20e8-4370-878f-1d7bc2f0222d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 12 17:48:09.137168 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:48:09.137092    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bd98bbd-20e8-4370-878f-1d7bc2f0222d-kube-api-access-sm2dg" (OuterVolumeSpecName: "kube-api-access-sm2dg") pod "9bd98bbd-20e8-4370-878f-1d7bc2f0222d" (UID: "9bd98bbd-20e8-4370-878f-1d7bc2f0222d"). InnerVolumeSpecName "kube-api-access-sm2dg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 12 17:48:09.236641 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:48:09.236617    2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9bd98bbd-20e8-4370-878f-1d7bc2f0222d-dshm\") on node \"ip-10-0-141-94.ec2.internal\" DevicePath \"\""
Jul 12 17:48:09.236641 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:48:09.236637    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sm2dg\" (UniqueName: \"kubernetes.io/projected/9bd98bbd-20e8-4370-878f-1d7bc2f0222d-kube-api-access-sm2dg\") on node \"ip-10-0-141-94.ec2.internal\" DevicePath \"\""
Jul 12 17:48:09.236769 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:48:09.236648    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9bd98bbd-20e8-4370-878f-1d7bc2f0222d-tls-certs\") on node \"ip-10-0-141-94.ec2.internal\" DevicePath \"\""
Jul 12 17:48:09.613214 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:48:09.613187    2569 generic.go:356] "Generic (PLEG): container finished" podID="9bd98bbd-20e8-4370-878f-1d7bc2f0222d" containerID="9630cb7080591e4be794833e587e317410f347a2cacf2543487e5ef9c625b274" exitCode=0
Jul 12 17:48:09.613382 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:48:09.613240    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-9c75b6/sub-test-model-9c75b6-kserve-69b5d55799-c4klq"
Jul 12 17:48:09.613382 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:48:09.613264    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-9c75b6/sub-test-model-9c75b6-kserve-69b5d55799-c4klq" event={"ID":"9bd98bbd-20e8-4370-878f-1d7bc2f0222d","Type":"ContainerDied","Data":"9630cb7080591e4be794833e587e317410f347a2cacf2543487e5ef9c625b274"}
Jul 12 17:48:09.613382 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:48:09.613308    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-9c75b6/sub-test-model-9c75b6-kserve-69b5d55799-c4klq" event={"ID":"9bd98bbd-20e8-4370-878f-1d7bc2f0222d","Type":"ContainerDied","Data":"a64b9b3b00088acbcd177764f7694bcc164d9a9376a382f5816072bbdc45213e"}
Jul 12 17:48:09.613382 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:48:09.613332    2569 scope.go:117] "RemoveContainer" containerID="9630cb7080591e4be794833e587e317410f347a2cacf2543487e5ef9c625b274"
Jul 12 17:48:09.625541 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:48:09.625501    2569 scope.go:117] "RemoveContainer" containerID="9630cb7080591e4be794833e587e317410f347a2cacf2543487e5ef9c625b274"
Jul 12 17:48:09.625820 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:48:09.625801    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9630cb7080591e4be794833e587e317410f347a2cacf2543487e5ef9c625b274\": container with ID starting with 9630cb7080591e4be794833e587e317410f347a2cacf2543487e5ef9c625b274 not found: ID does not exist" containerID="9630cb7080591e4be794833e587e317410f347a2cacf2543487e5ef9c625b274"
Jul 12 17:48:09.625897 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:48:09.625824    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9630cb7080591e4be794833e587e317410f347a2cacf2543487e5ef9c625b274"} err="failed to get container status \"9630cb7080591e4be794833e587e317410f347a2cacf2543487e5ef9c625b274\": rpc error: code = NotFound desc = could not find container \"9630cb7080591e4be794833e587e317410f347a2cacf2543487e5ef9c625b274\": container with ID starting with 9630cb7080591e4be794833e587e317410f347a2cacf2543487e5ef9c625b274 not found: ID does not exist"
Jul 12 17:48:09.636268 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:48:09.636247    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-9c75b6/sub-test-model-9c75b6-kserve-69b5d55799-c4klq"]
Jul 12 17:48:09.639393 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:48:09.639372    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-9c75b6/sub-test-model-9c75b6-kserve-69b5d55799-c4klq"]
Jul 12 17:48:10.379353 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:48:10.379324    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45af3f34-9698-4dcf-80f1-11131a96037e" path="/var/lib/kubelet/pods/45af3f34-9698-4dcf-80f1-11131a96037e/volumes"
Jul 12 17:48:10.379716 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:48:10.379698    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9bd98bbd-20e8-4370-878f-1d7bc2f0222d" path="/var/lib/kubelet/pods/9bd98bbd-20e8-4370-878f-1d7bc2f0222d/volumes"
Jul 12 17:48:12.468474 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:48:12.468448    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9xbrj_6b618491-4e81-4708-ab1b-2781d1df7c87/ovn-acl-logging/0.log"
Jul 12 17:48:12.473815 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:48:12.473793    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9xbrj_6b618491-4e81-4708-ab1b-2781d1df7c87/ovn-acl-logging/0.log"
Jul 12 17:48:39.690451 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:48:39.690411    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-9c75b6/rate-test-model-9c75b6-kserve-5c5f77465f-998lr"]
Jul 12 17:48:39.691020 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:48:39.690897    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9bd98bbd-20e8-4370-878f-1d7bc2f0222d" containerName="main"
Jul 12 17:48:39.691020 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:48:39.690916    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="9bd98bbd-20e8-4370-878f-1d7bc2f0222d" containerName="main"
Jul 12 17:48:39.691020 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:48:39.690940    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45af3f34-9698-4dcf-80f1-11131a96037e" containerName="main"
Jul 12 17:48:39.691020 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:48:39.690948    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="45af3f34-9698-4dcf-80f1-11131a96037e" containerName="main"
Jul 12 17:48:39.691219 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:48:39.691026    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="45af3f34-9698-4dcf-80f1-11131a96037e" containerName="main"
Jul 12 17:48:39.691219 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:48:39.691038    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="9bd98bbd-20e8-4370-878f-1d7bc2f0222d" containerName="main"
Jul 12 17:48:39.694073 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:48:39.694053    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-9c75b6/rate-test-model-9c75b6-kserve-5c5f77465f-998lr"
Jul 12 17:48:39.697200 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:48:39.697169    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-9c75b6\"/\"openshift-service-ca.crt\""
Jul 12 17:48:39.697311 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:48:39.697199    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-9c75b6\"/\"kube-root-ca.crt\""
Jul 12 17:48:39.697311 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:48:39.697219    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-9c75b6\"/\"default-dockercfg-n24tv\""
Jul 12 17:48:39.697311 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:48:39.697172    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-9c75b6\"/\"rate-test-model-9c75b6-kserve-self-signed-certs\""
Jul 12 17:48:39.702450 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:48:39.702428    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-9c75b6/rate-test-model-9c75b6-kserve-5c5f77465f-998lr"]
Jul 12 17:48:39.764394 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:48:39.764354    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19860435-9c7e-404b-9ae8-3792759fcf37-tmp-dir\") pod \"rate-test-model-9c75b6-kserve-5c5f77465f-998lr\" (UID: \"19860435-9c7e-404b-9ae8-3792759fcf37\") " pod="ai-tenant-e2e-shared-b-9c75b6/rate-test-model-9c75b6-kserve-5c5f77465f-998lr"
Jul 12 17:48:39.764394 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:48:39.764396    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19860435-9c7e-404b-9ae8-3792759fcf37-dshm\") pod \"rate-test-model-9c75b6-kserve-5c5f77465f-998lr\" (UID: \"19860435-9c7e-404b-9ae8-3792759fcf37\") " pod="ai-tenant-e2e-shared-b-9c75b6/rate-test-model-9c75b6-kserve-5c5f77465f-998lr"
Jul 12 17:48:39.764598 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:48:39.764458    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19860435-9c7e-404b-9ae8-3792759fcf37-home\") pod \"rate-test-model-9c75b6-kserve-5c5f77465f-998lr\" (UID: \"19860435-9c7e-404b-9ae8-3792759fcf37\") " pod="ai-tenant-e2e-shared-b-9c75b6/rate-test-model-9c75b6-kserve-5c5f77465f-998lr"
Jul 12 17:48:39.764598 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:48:39.764497    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rn2m5\" (UniqueName: \"kubernetes.io/projected/19860435-9c7e-404b-9ae8-3792759fcf37-kube-api-access-rn2m5\") pod \"rate-test-model-9c75b6-kserve-5c5f77465f-998lr\" (UID: \"19860435-9c7e-404b-9ae8-3792759fcf37\") " pod="ai-tenant-e2e-shared-b-9c75b6/rate-test-model-9c75b6-kserve-5c5f77465f-998lr"
Jul 12 17:48:39.764598 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:48:39.764550    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19860435-9c7e-404b-9ae8-3792759fcf37-model-cache\") pod \"rate-test-model-9c75b6-kserve-5c5f77465f-998lr\" (UID: \"19860435-9c7e-404b-9ae8-3792759fcf37\") " pod="ai-tenant-e2e-shared-b-9c75b6/rate-test-model-9c75b6-kserve-5c5f77465f-998lr"
Jul 12 17:48:39.764598 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:48:39.764592    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19860435-9c7e-404b-9ae8-3792759fcf37-tls-certs\") pod \"rate-test-model-9c75b6-kserve-5c5f77465f-998lr\" (UID: \"19860435-9c7e-404b-9ae8-3792759fcf37\") " pod="ai-tenant-e2e-shared-b-9c75b6/rate-test-model-9c75b6-kserve-5c5f77465f-998lr"
Jul 12 17:48:39.865962 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:48:39.865929    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19860435-9c7e-404b-9ae8-3792759fcf37-tmp-dir\") pod \"rate-test-model-9c75b6-kserve-5c5f77465f-998lr\" (UID: \"19860435-9c7e-404b-9ae8-3792759fcf37\") " pod="ai-tenant-e2e-shared-b-9c75b6/rate-test-model-9c75b6-kserve-5c5f77465f-998lr"
Jul 12 17:48:39.866123 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:48:39.865967    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19860435-9c7e-404b-9ae8-3792759fcf37-dshm\") pod \"rate-test-model-9c75b6-kserve-5c5f77465f-998lr\" (UID: \"19860435-9c7e-404b-9ae8-3792759fcf37\") " pod="ai-tenant-e2e-shared-b-9c75b6/rate-test-model-9c75b6-kserve-5c5f77465f-998lr"
Jul 12 17:48:39.866188 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:48:39.866138    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19860435-9c7e-404b-9ae8-3792759fcf37-home\") pod \"rate-test-model-9c75b6-kserve-5c5f77465f-998lr\" (UID: \"19860435-9c7e-404b-9ae8-3792759fcf37\") " pod="ai-tenant-e2e-shared-b-9c75b6/rate-test-model-9c75b6-kserve-5c5f77465f-998lr"
Jul 12 17:48:39.866243 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:48:39.866201    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rn2m5\" (UniqueName: \"kubernetes.io/projected/19860435-9c7e-404b-9ae8-3792759fcf37-kube-api-access-rn2m5\") pod \"rate-test-model-9c75b6-kserve-5c5f77465f-998lr\" (UID: \"19860435-9c7e-404b-9ae8-3792759fcf37\") " pod="ai-tenant-e2e-shared-b-9c75b6/rate-test-model-9c75b6-kserve-5c5f77465f-998lr"
Jul 12 17:48:39.866243 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:48:39.866233    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19860435-9c7e-404b-9ae8-3792759fcf37-model-cache\") pod \"rate-test-model-9c75b6-kserve-5c5f77465f-998lr\" (UID: \"19860435-9c7e-404b-9ae8-3792759fcf37\") " pod="ai-tenant-e2e-shared-b-9c75b6/rate-test-model-9c75b6-kserve-5c5f77465f-998lr"
Jul 12 17:48:39.866348 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:48:39.866285    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19860435-9c7e-404b-9ae8-3792759fcf37-tls-certs\") pod \"rate-test-model-9c75b6-kserve-5c5f77465f-998lr\" (UID: \"19860435-9c7e-404b-9ae8-3792759fcf37\") " pod="ai-tenant-e2e-shared-b-9c75b6/rate-test-model-9c75b6-kserve-5c5f77465f-998lr"
Jul 12 17:48:39.866451 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:48:39.866425    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19860435-9c7e-404b-9ae8-3792759fcf37-tmp-dir\") pod \"rate-test-model-9c75b6-kserve-5c5f77465f-998lr\" (UID: \"19860435-9c7e-404b-9ae8-3792759fcf37\") " pod="ai-tenant-e2e-shared-b-9c75b6/rate-test-model-9c75b6-kserve-5c5f77465f-998lr"
Jul 12 17:48:39.866608 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:48:39.866479    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19860435-9c7e-404b-9ae8-3792759fcf37-home\") pod \"rate-test-model-9c75b6-kserve-5c5f77465f-998lr\" (UID: \"19860435-9c7e-404b-9ae8-3792759fcf37\") " pod="ai-tenant-e2e-shared-b-9c75b6/rate-test-model-9c75b6-kserve-5c5f77465f-998lr"
Jul 12 17:48:39.866608 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:48:39.866603    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19860435-9c7e-404b-9ae8-3792759fcf37-model-cache\") pod \"rate-test-model-9c75b6-kserve-5c5f77465f-998lr\" (UID: \"19860435-9c7e-404b-9ae8-3792759fcf37\") " pod="ai-tenant-e2e-shared-b-9c75b6/rate-test-model-9c75b6-kserve-5c5f77465f-998lr"
Jul 12 17:48:39.868181 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:48:39.868157    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19860435-9c7e-404b-9ae8-3792759fcf37-dshm\") pod \"rate-test-model-9c75b6-kserve-5c5f77465f-998lr\" (UID: \"19860435-9c7e-404b-9ae8-3792759fcf37\") " pod="ai-tenant-e2e-shared-b-9c75b6/rate-test-model-9c75b6-kserve-5c5f77465f-998lr"
Jul 12 17:48:39.868590 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:48:39.868572    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19860435-9c7e-404b-9ae8-3792759fcf37-tls-certs\") pod \"rate-test-model-9c75b6-kserve-5c5f77465f-998lr\" (UID: \"19860435-9c7e-404b-9ae8-3792759fcf37\") " pod="ai-tenant-e2e-shared-b-9c75b6/rate-test-model-9c75b6-kserve-5c5f77465f-998lr"
Jul 12 17:48:39.873084 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:48:39.873060    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rn2m5\" (UniqueName: \"kubernetes.io/projected/19860435-9c7e-404b-9ae8-3792759fcf37-kube-api-access-rn2m5\") pod \"rate-test-model-9c75b6-kserve-5c5f77465f-998lr\" (UID: \"19860435-9c7e-404b-9ae8-3792759fcf37\") " pod="ai-tenant-e2e-shared-b-9c75b6/rate-test-model-9c75b6-kserve-5c5f77465f-998lr"
Jul 12 17:48:40.002713 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:48:40.002632    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-9c75b6/rate-test-model-9c75b6-kserve-5c5f77465f-998lr"
Jul 12 17:48:40.126439 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:48:40.126402    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-9c75b6/rate-test-model-9c75b6-kserve-5c5f77465f-998lr"]
Jul 12 17:48:40.129267 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:48:40.129238    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod19860435_9c7e_404b_9ae8_3792759fcf37.slice/crio-622c7c8332055dab7bf5169a7726e77e4996394b7e9263e5729a2e67238c535f WatchSource:0}: Error finding container 622c7c8332055dab7bf5169a7726e77e4996394b7e9263e5729a2e67238c535f: Status 404 returned error can't find the container with id 622c7c8332055dab7bf5169a7726e77e4996394b7e9263e5729a2e67238c535f
Jul 12 17:48:40.751461 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:48:40.751430    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-9c75b6/rate-test-model-9c75b6-kserve-5c5f77465f-998lr" event={"ID":"19860435-9c7e-404b-9ae8-3792759fcf37","Type":"ContainerStarted","Data":"ea2c78b9cce1bd743c0335638c01e81f1d468a058313b5723b1e88b186bb588c"}
Jul 12 17:48:40.751461 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:48:40.751467    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-9c75b6/rate-test-model-9c75b6-kserve-5c5f77465f-998lr" event={"ID":"19860435-9c7e-404b-9ae8-3792759fcf37","Type":"ContainerStarted","Data":"622c7c8332055dab7bf5169a7726e77e4996394b7e9263e5729a2e67238c535f"}
Jul 12 17:48:40.767175 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:48:40.767126    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-9c75b6/rate-test-model-9c75b6-kserve-5c5f77465f-998lr" podStartSLOduration=1.767112339 podStartE2EDuration="1.767112339s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 17:48:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 17:48:40.766403334 +0000 UTC m=+2728.886720154" watchObservedRunningTime="2026-07-12 17:48:40.767112339 +0000 UTC m=+2728.887429113"
Jul 12 17:48:43.991924 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:48:43.991894    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl4zn"]
Jul 12 17:48:50.003329 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:48:50.003296    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-9c75b6/rate-test-model-9c75b6-kserve-5c5f77465f-998lr"
Jul 12 17:48:50.003329 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:48:50.003336    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-9c75b6/rate-test-model-9c75b6-kserve-5c5f77465f-998lr"
Jul 12 17:48:50.015605 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:48:50.015576    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-9c75b6/rate-test-model-9c75b6-kserve-5c5f77465f-998lr"
Jul 12 17:48:50.805528 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:48:50.805486    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-9c75b6/rate-test-model-9c75b6-kserve-5c5f77465f-998lr"
Jul 12 17:49:11.887553 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:49:11.887479    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl4zn"]
Jul 12 17:49:30.487396 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:49:30.487364    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl4zn"]
Jul 12 17:49:35.079701 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:49:35.079617    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-e767e8/rate-test-model-e767e8-kserve-6895b4f675-q2qmr"]
Jul 12 17:49:35.972169 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:49:35.972136    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-9c75b6/rate-test-model-9c75b6-kserve-5c5f77465f-998lr"]
Jul 12 17:49:46.489705 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:49:46.489669    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl4zn"]
Jul 12 17:49:50.099489 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:49:50.099426    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-e767e8/rate-test-model-e767e8-kserve-6895b4f675-q2qmr" podUID="49e8743b-e8bb-4f9e-b498-7f700c469381" containerName="main" containerID="cri-o://2977ebfd3cef05a1a32025551589f9e09953d95fd25ea041dd86acfc86c00a59" gracePeriod=45
Jul 12 17:49:50.345839 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:49:50.345810    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-e767e8/rate-test-model-e767e8-kserve-6895b4f675-q2qmr"
Jul 12 17:49:50.433501 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:49:50.433477    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/49e8743b-e8bb-4f9e-b498-7f700c469381-dshm\") pod \"49e8743b-e8bb-4f9e-b498-7f700c469381\" (UID: \"49e8743b-e8bb-4f9e-b498-7f700c469381\") "
Jul 12 17:49:50.433660 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:49:50.433545    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/49e8743b-e8bb-4f9e-b498-7f700c469381-tls-certs\") pod \"49e8743b-e8bb-4f9e-b498-7f700c469381\" (UID: \"49e8743b-e8bb-4f9e-b498-7f700c469381\") "
Jul 12 17:49:50.433660 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:49:50.433589    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/49e8743b-e8bb-4f9e-b498-7f700c469381-home\") pod \"49e8743b-e8bb-4f9e-b498-7f700c469381\" (UID: \"49e8743b-e8bb-4f9e-b498-7f700c469381\") "
Jul 12 17:49:50.433660 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:49:50.433631    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/49e8743b-e8bb-4f9e-b498-7f700c469381-model-cache\") pod \"49e8743b-e8bb-4f9e-b498-7f700c469381\" (UID: \"49e8743b-e8bb-4f9e-b498-7f700c469381\") "
Jul 12 17:49:50.433829 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:49:50.433666    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ccrtg\" (UniqueName: \"kubernetes.io/projected/49e8743b-e8bb-4f9e-b498-7f700c469381-kube-api-access-ccrtg\") pod \"49e8743b-e8bb-4f9e-b498-7f700c469381\" (UID: \"49e8743b-e8bb-4f9e-b498-7f700c469381\") "
Jul 12 17:49:50.433829 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:49:50.433680    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49e8743b-e8bb-4f9e-b498-7f700c469381-tmp-dir\") pod \"49e8743b-e8bb-4f9e-b498-7f700c469381\" (UID: \"49e8743b-e8bb-4f9e-b498-7f700c469381\") "
Jul 12 17:49:50.433933 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:49:50.433813    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49e8743b-e8bb-4f9e-b498-7f700c469381-home" (OuterVolumeSpecName: "home") pod "49e8743b-e8bb-4f9e-b498-7f700c469381" (UID: "49e8743b-e8bb-4f9e-b498-7f700c469381"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 12 17:49:50.433933 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:49:50.433905    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49e8743b-e8bb-4f9e-b498-7f700c469381-model-cache" (OuterVolumeSpecName: "model-cache") pod "49e8743b-e8bb-4f9e-b498-7f700c469381" (UID: "49e8743b-e8bb-4f9e-b498-7f700c469381"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 12 17:49:50.434038 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:49:50.433937    2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/49e8743b-e8bb-4f9e-b498-7f700c469381-home\") on node \"ip-10-0-141-94.ec2.internal\" DevicePath \"\""
Jul 12 17:49:50.434094 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:49:50.434040    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49e8743b-e8bb-4f9e-b498-7f700c469381-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "49e8743b-e8bb-4f9e-b498-7f700c469381" (UID: "49e8743b-e8bb-4f9e-b498-7f700c469381"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 12 17:49:50.435403 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:49:50.435383    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49e8743b-e8bb-4f9e-b498-7f700c469381-dshm" (OuterVolumeSpecName: "dshm") pod "49e8743b-e8bb-4f9e-b498-7f700c469381" (UID: "49e8743b-e8bb-4f9e-b498-7f700c469381"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 12 17:49:50.435520 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:49:50.435435    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49e8743b-e8bb-4f9e-b498-7f700c469381-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "49e8743b-e8bb-4f9e-b498-7f700c469381" (UID: "49e8743b-e8bb-4f9e-b498-7f700c469381"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 12 17:49:50.435597 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:49:50.435576    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49e8743b-e8bb-4f9e-b498-7f700c469381-kube-api-access-ccrtg" (OuterVolumeSpecName: "kube-api-access-ccrtg") pod "49e8743b-e8bb-4f9e-b498-7f700c469381" (UID: "49e8743b-e8bb-4f9e-b498-7f700c469381"). InnerVolumeSpecName "kube-api-access-ccrtg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 12 17:49:50.535109 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:49:50.535077    2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/49e8743b-e8bb-4f9e-b498-7f700c469381-model-cache\") on node \"ip-10-0-141-94.ec2.internal\" DevicePath \"\""
Jul 12 17:49:50.535109 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:49:50.535109    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ccrtg\" (UniqueName: \"kubernetes.io/projected/49e8743b-e8bb-4f9e-b498-7f700c469381-kube-api-access-ccrtg\") on node \"ip-10-0-141-94.ec2.internal\" DevicePath \"\""
Jul 12 17:49:50.535236 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:49:50.535120    2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49e8743b-e8bb-4f9e-b498-7f700c469381-tmp-dir\") on node \"ip-10-0-141-94.ec2.internal\" DevicePath \"\""
Jul 12 17:49:50.535236 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:49:50.535128    2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/49e8743b-e8bb-4f9e-b498-7f700c469381-dshm\") on node \"ip-10-0-141-94.ec2.internal\" DevicePath \"\""
Jul 12 17:49:50.535236 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:49:50.535135    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/49e8743b-e8bb-4f9e-b498-7f700c469381-tls-certs\") on node \"ip-10-0-141-94.ec2.internal\" DevicePath \"\""
Jul 12 17:49:50.992148 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:49:50.992080    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-9c75b6/rate-test-model-9c75b6-kserve-5c5f77465f-998lr" podUID="19860435-9c7e-404b-9ae8-3792759fcf37" containerName="main" containerID="cri-o://ea2c78b9cce1bd743c0335638c01e81f1d468a058313b5723b1e88b186bb588c" gracePeriod=45
Jul 12 17:49:51.051338 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:49:51.051304    2569 generic.go:356] "Generic (PLEG): container finished" podID="49e8743b-e8bb-4f9e-b498-7f700c469381" containerID="2977ebfd3cef05a1a32025551589f9e09953d95fd25ea041dd86acfc86c00a59" exitCode=0
Jul 12 17:49:51.051478 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:49:51.051340    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-e767e8/rate-test-model-e767e8-kserve-6895b4f675-q2qmr" event={"ID":"49e8743b-e8bb-4f9e-b498-7f700c469381","Type":"ContainerDied","Data":"2977ebfd3cef05a1a32025551589f9e09953d95fd25ea041dd86acfc86c00a59"}
Jul 12 17:49:51.051478 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:49:51.051393    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-e767e8/rate-test-model-e767e8-kserve-6895b4f675-q2qmr" event={"ID":"49e8743b-e8bb-4f9e-b498-7f700c469381","Type":"ContainerDied","Data":"a0d83d87e4b35fe310217fac302310a7294a82ed8b505edccaaf92bdcd9f4900"}
Jul 12 17:49:51.051478 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:49:51.051412    2569 scope.go:117] "RemoveContainer" containerID="2977ebfd3cef05a1a32025551589f9e09953d95fd25ea041dd86acfc86c00a59"
Jul 12 17:49:51.051478 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:49:51.051356    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-e767e8/rate-test-model-e767e8-kserve-6895b4f675-q2qmr"
Jul 12 17:49:51.069964 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:49:51.069943    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-e767e8/rate-test-model-e767e8-kserve-6895b4f675-q2qmr"]
Jul 12 17:49:51.073810 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:49:51.073791    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-e767e8/rate-test-model-e767e8-kserve-6895b4f675-q2qmr"]
Jul 12 17:49:51.102099 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:49:51.101961    2569 scope.go:117] "RemoveContainer" containerID="2977ebfd3cef05a1a32025551589f9e09953d95fd25ea041dd86acfc86c00a59"
Jul 12 17:49:51.102319 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:49:51.102237    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2977ebfd3cef05a1a32025551589f9e09953d95fd25ea041dd86acfc86c00a59\": container with ID starting with 2977ebfd3cef05a1a32025551589f9e09953d95fd25ea041dd86acfc86c00a59 not found: ID does not exist" containerID="2977ebfd3cef05a1a32025551589f9e09953d95fd25ea041dd86acfc86c00a59"
Jul 12 17:49:51.102319 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:49:51.102262    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2977ebfd3cef05a1a32025551589f9e09953d95fd25ea041dd86acfc86c00a59"} err="failed to get container status \"2977ebfd3cef05a1a32025551589f9e09953d95fd25ea041dd86acfc86c00a59\": rpc error: code = NotFound desc = could not find container \"2977ebfd3cef05a1a32025551589f9e09953d95fd25ea041dd86acfc86c00a59\": container with ID starting with 2977ebfd3cef05a1a32025551589f9e09953d95fd25ea041dd86acfc86c00a59 not found: ID does not exist"
Jul 12 17:49:51.247468 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:49:51.247410    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-9c75b6/rate-test-model-9c75b6-kserve-5c5f77465f-998lr"
Jul 12 17:49:51.342862 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:49:51.342833    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19860435-9c7e-404b-9ae8-3792759fcf37-model-cache\") pod \"19860435-9c7e-404b-9ae8-3792759fcf37\" (UID: \"19860435-9c7e-404b-9ae8-3792759fcf37\") "
Jul 12 17:49:51.343009 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:49:51.342872    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19860435-9c7e-404b-9ae8-3792759fcf37-tmp-dir\") pod \"19860435-9c7e-404b-9ae8-3792759fcf37\" (UID: \"19860435-9c7e-404b-9ae8-3792759fcf37\") "
Jul 12 17:49:51.343009 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:49:51.342891    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19860435-9c7e-404b-9ae8-3792759fcf37-home\") pod \"19860435-9c7e-404b-9ae8-3792759fcf37\" (UID: \"19860435-9c7e-404b-9ae8-3792759fcf37\") "
Jul 12 17:49:51.343009 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:49:51.342913    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19860435-9c7e-404b-9ae8-3792759fcf37-tls-certs\") pod \"19860435-9c7e-404b-9ae8-3792759fcf37\" (UID: \"19860435-9c7e-404b-9ae8-3792759fcf37\") "
Jul 12 17:49:51.343009 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:49:51.342956    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19860435-9c7e-404b-9ae8-3792759fcf37-dshm\") pod \"19860435-9c7e-404b-9ae8-3792759fcf37\" (UID: \"19860435-9c7e-404b-9ae8-3792759fcf37\") "
Jul 12 17:49:51.343009 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:49:51.343002    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rn2m5\" (UniqueName: \"kubernetes.io/projected/19860435-9c7e-404b-9ae8-3792759fcf37-kube-api-access-rn2m5\") pod \"19860435-9c7e-404b-9ae8-3792759fcf37\" (UID: \"19860435-9c7e-404b-9ae8-3792759fcf37\") "
Jul 12 17:49:51.343256 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:49:51.343136    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19860435-9c7e-404b-9ae8-3792759fcf37-home" (OuterVolumeSpecName: "home") pod "19860435-9c7e-404b-9ae8-3792759fcf37" (UID: "19860435-9c7e-404b-9ae8-3792759fcf37"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 12 17:49:51.343256 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:49:51.343145    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19860435-9c7e-404b-9ae8-3792759fcf37-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "19860435-9c7e-404b-9ae8-3792759fcf37" (UID: "19860435-9c7e-404b-9ae8-3792759fcf37"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 12 17:49:51.343256 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:49:51.343159    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19860435-9c7e-404b-9ae8-3792759fcf37-model-cache" (OuterVolumeSpecName: "model-cache") pod "19860435-9c7e-404b-9ae8-3792759fcf37" (UID: "19860435-9c7e-404b-9ae8-3792759fcf37"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 12 17:49:51.343256 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:49:51.343245    2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19860435-9c7e-404b-9ae8-3792759fcf37-model-cache\") on node \"ip-10-0-141-94.ec2.internal\" DevicePath \"\""
Jul 12 17:49:51.343390 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:49:51.343261    2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19860435-9c7e-404b-9ae8-3792759fcf37-tmp-dir\") on node \"ip-10-0-141-94.ec2.internal\" DevicePath \"\""
Jul 12 17:49:51.343390 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:49:51.343273    2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19860435-9c7e-404b-9ae8-3792759fcf37-home\") on node \"ip-10-0-141-94.ec2.internal\" DevicePath \"\""
Jul 12 17:49:51.344959 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:49:51.344937    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19860435-9c7e-404b-9ae8-3792759fcf37-kube-api-access-rn2m5" (OuterVolumeSpecName: "kube-api-access-rn2m5") pod "19860435-9c7e-404b-9ae8-3792759fcf37" (UID: "19860435-9c7e-404b-9ae8-3792759fcf37"). InnerVolumeSpecName "kube-api-access-rn2m5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 12 17:49:51.345051 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:49:51.344981    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19860435-9c7e-404b-9ae8-3792759fcf37-dshm" (OuterVolumeSpecName: "dshm") pod "19860435-9c7e-404b-9ae8-3792759fcf37" (UID: "19860435-9c7e-404b-9ae8-3792759fcf37"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 12 17:49:51.345051 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:49:51.345023    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19860435-9c7e-404b-9ae8-3792759fcf37-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "19860435-9c7e-404b-9ae8-3792759fcf37" (UID: "19860435-9c7e-404b-9ae8-3792759fcf37"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 12 17:49:51.444516 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:49:51.444490    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rn2m5\" (UniqueName: \"kubernetes.io/projected/19860435-9c7e-404b-9ae8-3792759fcf37-kube-api-access-rn2m5\") on node \"ip-10-0-141-94.ec2.internal\" DevicePath \"\""
Jul 12 17:49:51.444627 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:49:51.444527    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19860435-9c7e-404b-9ae8-3792759fcf37-tls-certs\") on node \"ip-10-0-141-94.ec2.internal\" DevicePath \"\""
Jul 12 17:49:51.444627 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:49:51.444537    2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19860435-9c7e-404b-9ae8-3792759fcf37-dshm\") on node \"ip-10-0-141-94.ec2.internal\" DevicePath \"\""
Jul 12 17:49:52.056471 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:49:52.056436    2569 generic.go:356] "Generic (PLEG): container finished" podID="19860435-9c7e-404b-9ae8-3792759fcf37" containerID="ea2c78b9cce1bd743c0335638c01e81f1d468a058313b5723b1e88b186bb588c" exitCode=0
Jul 12 17:49:52.056777 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:49:52.056481    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-9c75b6/rate-test-model-9c75b6-kserve-5c5f77465f-998lr"
Jul 12 17:49:52.056777 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:49:52.056542    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-9c75b6/rate-test-model-9c75b6-kserve-5c5f77465f-998lr" event={"ID":"19860435-9c7e-404b-9ae8-3792759fcf37","Type":"ContainerDied","Data":"ea2c78b9cce1bd743c0335638c01e81f1d468a058313b5723b1e88b186bb588c"}
Jul 12 17:49:52.056777 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:49:52.056590    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-9c75b6/rate-test-model-9c75b6-kserve-5c5f77465f-998lr" event={"ID":"19860435-9c7e-404b-9ae8-3792759fcf37","Type":"ContainerDied","Data":"622c7c8332055dab7bf5169a7726e77e4996394b7e9263e5729a2e67238c535f"}
Jul 12 17:49:52.056777 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:49:52.056610    2569 scope.go:117] "RemoveContainer" containerID="ea2c78b9cce1bd743c0335638c01e81f1d468a058313b5723b1e88b186bb588c"
Jul 12 17:49:52.065277 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:49:52.065261    2569 scope.go:117] "RemoveContainer" containerID="ea2c78b9cce1bd743c0335638c01e81f1d468a058313b5723b1e88b186bb588c"
Jul 12 17:49:52.065470 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:49:52.065454    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea2c78b9cce1bd743c0335638c01e81f1d468a058313b5723b1e88b186bb588c\": container with ID starting with ea2c78b9cce1bd743c0335638c01e81f1d468a058313b5723b1e88b186bb588c not found: ID does not exist" containerID="ea2c78b9cce1bd743c0335638c01e81f1d468a058313b5723b1e88b186bb588c"
Jul 12 17:49:52.065543 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:49:52.065477    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea2c78b9cce1bd743c0335638c01e81f1d468a058313b5723b1e88b186bb588c"} err="failed to get container status \"ea2c78b9cce1bd743c0335638c01e81f1d468a058313b5723b1e88b186bb588c\": rpc error: code = NotFound desc = could not find container \"ea2c78b9cce1bd743c0335638c01e81f1d468a058313b5723b1e88b186bb588c\": container with ID starting with ea2c78b9cce1bd743c0335638c01e81f1d468a058313b5723b1e88b186bb588c not found: ID does not exist"
Jul 12 17:49:52.075454 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:49:52.075431    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-9c75b6/rate-test-model-9c75b6-kserve-5c5f77465f-998lr"]
Jul 12 17:49:52.078931 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:49:52.078909    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-9c75b6/rate-test-model-9c75b6-kserve-5c5f77465f-998lr"]
Jul 12 17:49:52.379620 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:49:52.379541    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="19860435-9c7e-404b-9ae8-3792759fcf37" path="/var/lib/kubelet/pods/19860435-9c7e-404b-9ae8-3792759fcf37/volumes"
Jul 12 17:49:52.380097 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:49:52.380074    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="49e8743b-e8bb-4f9e-b498-7f700c469381" path="/var/lib/kubelet/pods/49e8743b-e8bb-4f9e-b498-7f700c469381/volumes"
Jul 12 17:49:52.678897 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:49:52.678865    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7477-140060958484784"]
Jul 12 17:49:52.679224 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:49:52.679208    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19860435-9c7e-404b-9ae8-3792759fcf37" containerName="main"
Jul 12 17:49:52.679303 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:49:52.679226    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="19860435-9c7e-404b-9ae8-3792759fcf37" containerName="main"
Jul 12 17:49:52.679303 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:49:52.679257    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49e8743b-e8bb-4f9e-b498-7f700c469381" containerName="main"
Jul 12 17:49:52.679303 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:49:52.679266    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="49e8743b-e8bb-4f9e-b498-7f700c469381" containerName="main"
Jul 12 17:49:52.679448 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:49:52.679341    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="19860435-9c7e-404b-9ae8-3792759fcf37" containerName="main"
Jul 12 17:49:52.679448 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:49:52.679355    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="49e8743b-e8bb-4f9e-b498-7f700c469381" containerName="main"
Jul 12 17:49:52.683646 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:49:52.683624    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7477-140060958484784"
Jul 12 17:49:52.685549 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:49:52.685527    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7477-140060958484784"]
Jul 12 17:49:52.686009 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:49:52.685988    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-dockercfg-vv9ps\""
Jul 12 17:49:52.856818 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:49:52.856791    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbrkq\" (UniqueName: \"kubernetes.io/projected/78e1c14b-bce6-448a-a604-18a40a7f75f0-kube-api-access-dbrkq\") pod \"test-curl-7477-140060958484784\" (UID: \"78e1c14b-bce6-448a-a604-18a40a7f75f0\") " pod="openshift-ingress/test-curl-7477-140060958484784"
Jul 12 17:49:52.958217 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:49:52.958147    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbrkq\" (UniqueName: \"kubernetes.io/projected/78e1c14b-bce6-448a-a604-18a40a7f75f0-kube-api-access-dbrkq\") pod \"test-curl-7477-140060958484784\" (UID: \"78e1c14b-bce6-448a-a604-18a40a7f75f0\") " pod="openshift-ingress/test-curl-7477-140060958484784"
Jul 12 17:49:52.965819 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:49:52.965795    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbrkq\" (UniqueName: \"kubernetes.io/projected/78e1c14b-bce6-448a-a604-18a40a7f75f0-kube-api-access-dbrkq\") pod \"test-curl-7477-140060958484784\" (UID: \"78e1c14b-bce6-448a-a604-18a40a7f75f0\") " pod="openshift-ingress/test-curl-7477-140060958484784"
Jul 12 17:49:52.993067 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:49:52.993045    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7477-140060958484784"
Jul 12 17:49:53.115097 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:49:53.115046    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7477-140060958484784"]
Jul 12 17:49:53.117342 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:49:53.117314    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod78e1c14b_bce6_448a_a604_18a40a7f75f0.slice/crio-ed099e10045fd4a3f23e5bd64de78d9cb6e4f467ff57438729a5ddcc91331151 WatchSource:0}: Error finding container ed099e10045fd4a3f23e5bd64de78d9cb6e4f467ff57438729a5ddcc91331151: Status 404 returned error can't find the container with id ed099e10045fd4a3f23e5bd64de78d9cb6e4f467ff57438729a5ddcc91331151
Jul 12 17:49:53.118627 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:49:53.118604    2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 12 17:49:54.068225 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:49:54.068146    2569 generic.go:356] "Generic (PLEG): container finished" podID="78e1c14b-bce6-448a-a604-18a40a7f75f0" containerID="0db60420a8210d6404badd2160cca163b6efa95a5813dc657913c182cd0257af" exitCode=0
Jul 12 17:49:54.068602 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:49:54.068237    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7477-140060958484784" event={"ID":"78e1c14b-bce6-448a-a604-18a40a7f75f0","Type":"ContainerDied","Data":"0db60420a8210d6404badd2160cca163b6efa95a5813dc657913c182cd0257af"}
Jul 12 17:49:54.068602 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:49:54.068276    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7477-140060958484784" event={"ID":"78e1c14b-bce6-448a-a604-18a40a7f75f0","Type":"ContainerStarted","Data":"ed099e10045fd4a3f23e5bd64de78d9cb6e4f467ff57438729a5ddcc91331151"}
Jul 12 17:49:55.206143 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:49:55.206114    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7477-140060958484784"
Jul 12 17:49:55.238757 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:49:55.238725    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7477-140060958484784_78e1c14b-bce6-448a-a604-18a40a7f75f0/test-curl-7477-140060958484784/0.log"
Jul 12 17:49:55.262895 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:49:55.262865    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7477-140060958484784"]
Jul 12 17:49:55.266787 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:49:55.266764    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7477-140060958484784"]
Jul 12 17:49:55.378713 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:49:55.378636    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dbrkq\" (UniqueName: \"kubernetes.io/projected/78e1c14b-bce6-448a-a604-18a40a7f75f0-kube-api-access-dbrkq\") pod \"78e1c14b-bce6-448a-a604-18a40a7f75f0\" (UID: \"78e1c14b-bce6-448a-a604-18a40a7f75f0\") "
Jul 12 17:49:55.380587 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:49:55.380561    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78e1c14b-bce6-448a-a604-18a40a7f75f0-kube-api-access-dbrkq" (OuterVolumeSpecName: "kube-api-access-dbrkq") pod "78e1c14b-bce6-448a-a604-18a40a7f75f0" (UID: "78e1c14b-bce6-448a-a604-18a40a7f75f0"). InnerVolumeSpecName "kube-api-access-dbrkq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 12 17:49:55.479885 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:49:55.479849    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbrkq\" (UniqueName: \"kubernetes.io/projected/78e1c14b-bce6-448a-a604-18a40a7f75f0-kube-api-access-dbrkq\") on node \"ip-10-0-141-94.ec2.internal\" DevicePath \"\""
Jul 12 17:49:56.078036 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:49:56.078008    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7477-140060958484784"
Jul 12 17:49:56.078036 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:49:56.078029    2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ed099e10045fd4a3f23e5bd64de78d9cb6e4f467ff57438729a5ddcc91331151"
Jul 12 17:49:56.087045 ip-10-0-141-94 kubenswrapper[2569]: E0712 17:49:56.087017    2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7477-140060958484784\" is forbidden: User \"system:node:ip-10-0-141-94.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-141-94.ec2.internal' and this object" podUID="78e1c14b-bce6-448a-a604-18a40a7f75f0" pod="openshift-ingress/test-curl-7477-140060958484784"
Jul 12 17:49:56.379377 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:49:56.379310    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="78e1c14b-bce6-448a-a604-18a40a7f75f0" path="/var/lib/kubelet/pods/78e1c14b-bce6-448a-a604-18a40a7f75f0/volumes"
Jul 12 17:50:09.385251 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:50:09.385200    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl4zn"]
Jul 12 17:50:33.687694 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:50:33.687657    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl4zn"]
Jul 12 17:50:54.105413 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:50:54.105382    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-554bd9d4cc-b2lhx_1aa36e66-d3fd-4083-b35a-e62fefcacdd9/copy-manifests/0.log"
Jul 12 17:50:54.111820 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:50:54.111799    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-554bd9d4cc-b2lhx_1aa36e66-d3fd-4083-b35a-e62fefcacdd9/manager/0.log"
Jul 12 17:50:57.867939 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:50:57.867911    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-cmbk5_9b68d9a9-6bc7-4f78-a3dd-b5a56a5aa7d4/kuadrant-console-plugin/0.log"
Jul 12 17:50:58.270619 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:50:58.270590    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-hl4zn_3715331f-8a45-4780-9bfe-ceae9bea46ec/limitador/0.log"
Jul 12 17:50:58.952971 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:50:58.952937    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-59584446bf-bxv2x_327a7007-ca31-49a8-acda-91bf35ed6963/kube-auth-proxy/0.log"
Jul 12 17:50:59.868457 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:50:59.868421    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-4ldrp_994b775c-7cf7-449e-9f40-17e7b9afa9b5/main/0.log"
Jul 12 17:51:00.305470 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:51:00.305429    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-nvspc_981e1044-8c89-4da6-8476-6828123a6efc/main/0.log"
Jul 12 17:51:00.412794 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:51:00.412766    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5fc6fb456f-tc285_f5cf5611-64a5-4ace-a2c2-16d2d52b32da/main/0.log"
Jul 12 17:51:07.388356 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:51:07.388266    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-mkw2b_577e3a93-0560-4099-bddb-dd55563d002d/global-pull-secret-syncer/0.log"
Jul 12 17:51:07.589031 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:51:07.588990    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-kjdjh_5598dd9a-714f-4e7d-9c59-a6e2f61c5a74/konnectivity-agent/0.log"
Jul 12 17:51:07.655342 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:51:07.655268    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-94.ec2.internal_64598ecc4535f6658a01b7cab4f3caeb/haproxy/0.log"
Jul 12 17:51:13.203178 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:51:13.203148    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-cmbk5_9b68d9a9-6bc7-4f78-a3dd-b5a56a5aa7d4/kuadrant-console-plugin/0.log"
Jul 12 17:51:13.513057 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:51:13.513027    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-hl4zn_3715331f-8a45-4780-9bfe-ceae9bea46ec/limitador/0.log"
Jul 12 17:51:15.545200 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:51:15.545170    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-5866f48f55-vmnx5_63874cd4-7f4d-4834-9e6b-1e524494498a/metrics-server/0.log"
Jul 12 17:51:15.612126 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:51:15.612091    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5f68df7f58-862s7_dcf4607e-6cb3-4f9c-b3b3-9e2245bc8867/monitoring-plugin/0.log"
Jul 12 17:51:15.674733 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:51:15.674702    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gc7xw_943fb073-b5af-467c-a4fe-16d9a0cb07bc/node-exporter/0.log"
Jul 12 17:51:15.701709 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:51:15.701682    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gc7xw_943fb073-b5af-467c-a4fe-16d9a0cb07bc/kube-rbac-proxy/0.log"
Jul 12 17:51:15.721638 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:51:15.721615    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gc7xw_943fb073-b5af-467c-a4fe-16d9a0cb07bc/init-textfile/0.log"
Jul 12 17:51:15.891998 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:51:15.891888    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-x46rv_156026b1-c28b-43ca-9249-6c85d5de032e/kube-rbac-proxy-main/0.log"
Jul 12 17:51:15.913029 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:51:15.912998    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-x46rv_156026b1-c28b-43ca-9249-6c85d5de032e/kube-rbac-proxy-self/0.log"
Jul 12 17:51:15.938683 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:51:15.938655    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-x46rv_156026b1-c28b-43ca-9249-6c85d5de032e/openshift-state-metrics/0.log"
Jul 12 17:51:15.985293 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:51:15.985263    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_88eea35c-c9d1-4c86-8289-9fba5f31f6fd/prometheus/0.log"
Jul 12 17:51:16.004571 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:51:16.004545    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_88eea35c-c9d1-4c86-8289-9fba5f31f6fd/config-reloader/0.log"
Jul 12 17:51:16.023539 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:51:16.023496    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_88eea35c-c9d1-4c86-8289-9fba5f31f6fd/thanos-sidecar/0.log"
Jul 12 17:51:16.044357 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:51:16.044322    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_88eea35c-c9d1-4c86-8289-9fba5f31f6fd/kube-rbac-proxy-web/0.log"
Jul 12 17:51:16.067799 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:51:16.067773    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_88eea35c-c9d1-4c86-8289-9fba5f31f6fd/kube-rbac-proxy/0.log"
Jul 12 17:51:16.086106 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:51:16.086076    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_88eea35c-c9d1-4c86-8289-9fba5f31f6fd/kube-rbac-proxy-thanos/0.log"
Jul 12 17:51:16.104745 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:51:16.104688    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_88eea35c-c9d1-4c86-8289-9fba5f31f6fd/init-config-reloader/0.log"
Jul 12 17:51:16.129743 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:51:16.129712    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-64785b6675-jkqx9_23b6cfce-b79a-4c63-9c76-1932981d702b/prometheus-operator/0.log"
Jul 12 17:51:16.147693 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:51:16.147611    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-64785b6675-jkqx9_23b6cfce-b79a-4c63-9c76-1932981d702b/kube-rbac-proxy/0.log"
Jul 12 17:51:17.779485 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:51:17.779453    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-rpjth/perf-node-gather-daemonset-hnccj"]
Jul 12 17:51:17.779844 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:51:17.779777    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78e1c14b-bce6-448a-a604-18a40a7f75f0" containerName="test-curl-7477-140060958484784"
Jul 12 17:51:17.779844 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:51:17.779789    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="78e1c14b-bce6-448a-a604-18a40a7f75f0" containerName="test-curl-7477-140060958484784"
Jul 12 17:51:17.779844 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:51:17.779838    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="78e1c14b-bce6-448a-a604-18a40a7f75f0" containerName="test-curl-7477-140060958484784"
Jul 12 17:51:17.783036 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:51:17.783010    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rpjth/perf-node-gather-daemonset-hnccj"
Jul 12 17:51:17.785102 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:51:17.785082    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rpjth\"/\"kube-root-ca.crt\""
Jul 12 17:51:17.786055 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:51:17.786028    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rpjth\"/\"openshift-service-ca.crt\""
Jul 12 17:51:17.786184 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:51:17.786028    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-rpjth\"/\"default-dockercfg-pfmzn\""
Jul 12 17:51:17.791762 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:51:17.791741    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rpjth/perf-node-gather-daemonset-hnccj"]
Jul 12 17:51:17.809182 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:51:17.809159    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/be0a1fb9-5e64-4622-abea-d03eae829bea-lib-modules\") pod \"perf-node-gather-daemonset-hnccj\" (UID: \"be0a1fb9-5e64-4622-abea-d03eae829bea\") " pod="openshift-must-gather-rpjth/perf-node-gather-daemonset-hnccj"
Jul 12 17:51:17.809312 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:51:17.809189    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/be0a1fb9-5e64-4622-abea-d03eae829bea-podres\") pod \"perf-node-gather-daemonset-hnccj\" (UID: \"be0a1fb9-5e64-4622-abea-d03eae829bea\") " pod="openshift-must-gather-rpjth/perf-node-gather-daemonset-hnccj"
Jul 12 17:51:17.809312 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:51:17.809242    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/be0a1fb9-5e64-4622-abea-d03eae829bea-proc\") pod \"perf-node-gather-daemonset-hnccj\" (UID: \"be0a1fb9-5e64-4622-abea-d03eae829bea\") " pod="openshift-must-gather-rpjth/perf-node-gather-daemonset-hnccj"
Jul 12 17:51:17.809389 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:51:17.809313    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/be0a1fb9-5e64-4622-abea-d03eae829bea-sys\") pod \"perf-node-gather-daemonset-hnccj\" (UID: \"be0a1fb9-5e64-4622-abea-d03eae829bea\") " pod="openshift-must-gather-rpjth/perf-node-gather-daemonset-hnccj"
Jul 12 17:51:17.809389 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:51:17.809332    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5gb6\" (UniqueName: \"kubernetes.io/projected/be0a1fb9-5e64-4622-abea-d03eae829bea-kube-api-access-f5gb6\") pod \"perf-node-gather-daemonset-hnccj\" (UID: \"be0a1fb9-5e64-4622-abea-d03eae829bea\") " pod="openshift-must-gather-rpjth/perf-node-gather-daemonset-hnccj"
Jul 12 17:51:17.909800 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:51:17.909778    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/be0a1fb9-5e64-4622-abea-d03eae829bea-proc\") pod \"perf-node-gather-daemonset-hnccj\" (UID: \"be0a1fb9-5e64-4622-abea-d03eae829bea\") " pod="openshift-must-gather-rpjth/perf-node-gather-daemonset-hnccj"
Jul 12 17:51:17.909911 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:51:17.909832    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/be0a1fb9-5e64-4622-abea-d03eae829bea-sys\") pod \"perf-node-gather-daemonset-hnccj\" (UID: \"be0a1fb9-5e64-4622-abea-d03eae829bea\") " pod="openshift-must-gather-rpjth/perf-node-gather-daemonset-hnccj"
Jul 12 17:51:17.909911 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:51:17.909856    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5gb6\" (UniqueName: \"kubernetes.io/projected/be0a1fb9-5e64-4622-abea-d03eae829bea-kube-api-access-f5gb6\") pod \"perf-node-gather-daemonset-hnccj\" (UID: \"be0a1fb9-5e64-4622-abea-d03eae829bea\") " pod="openshift-must-gather-rpjth/perf-node-gather-daemonset-hnccj"
Jul 12 17:51:17.909911 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:51:17.909878    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/be0a1fb9-5e64-4622-abea-d03eae829bea-lib-modules\") pod \"perf-node-gather-daemonset-hnccj\" (UID: \"be0a1fb9-5e64-4622-abea-d03eae829bea\") " pod="openshift-must-gather-rpjth/perf-node-gather-daemonset-hnccj"
Jul 12 17:51:17.909911 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:51:17.909894    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/be0a1fb9-5e64-4622-abea-d03eae829bea-podres\") pod \"perf-node-gather-daemonset-hnccj\" (UID: \"be0a1fb9-5e64-4622-abea-d03eae829bea\") " pod="openshift-must-gather-rpjth/perf-node-gather-daemonset-hnccj"
Jul 12 17:51:17.910095 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:51:17.909893    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/be0a1fb9-5e64-4622-abea-d03eae829bea-proc\") pod \"perf-node-gather-daemonset-hnccj\" (UID: \"be0a1fb9-5e64-4622-abea-d03eae829bea\") " pod="openshift-must-gather-rpjth/perf-node-gather-daemonset-hnccj"
Jul 12 17:51:17.910095 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:51:17.909963    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/be0a1fb9-5e64-4622-abea-d03eae829bea-sys\") pod \"perf-node-gather-daemonset-hnccj\" (UID: \"be0a1fb9-5e64-4622-abea-d03eae829bea\") " pod="openshift-must-gather-rpjth/perf-node-gather-daemonset-hnccj"
Jul 12 17:51:17.910095 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:51:17.910016    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/be0a1fb9-5e64-4622-abea-d03eae829bea-podres\") pod \"perf-node-gather-daemonset-hnccj\" (UID: \"be0a1fb9-5e64-4622-abea-d03eae829bea\") " pod="openshift-must-gather-rpjth/perf-node-gather-daemonset-hnccj"
Jul 12 17:51:17.910095 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:51:17.910025    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/be0a1fb9-5e64-4622-abea-d03eae829bea-lib-modules\") pod \"perf-node-gather-daemonset-hnccj\" (UID: \"be0a1fb9-5e64-4622-abea-d03eae829bea\") " pod="openshift-must-gather-rpjth/perf-node-gather-daemonset-hnccj"
Jul 12 17:51:17.917831 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:51:17.917812    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5gb6\" (UniqueName: \"kubernetes.io/projected/be0a1fb9-5e64-4622-abea-d03eae829bea-kube-api-access-f5gb6\") pod \"perf-node-gather-daemonset-hnccj\" (UID: \"be0a1fb9-5e64-4622-abea-d03eae829bea\") " pod="openshift-must-gather-rpjth/perf-node-gather-daemonset-hnccj"
Jul 12 17:51:18.093584 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:51:18.093497    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rpjth/perf-node-gather-daemonset-hnccj"
Jul 12 17:51:18.217005 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:51:18.216923    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rpjth/perf-node-gather-daemonset-hnccj"]
Jul 12 17:51:18.219640 ip-10-0-141-94 kubenswrapper[2569]: W0712 17:51:18.219611    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podbe0a1fb9_5e64_4622_abea_d03eae829bea.slice/crio-252aafc3b37206597fa016aceb9e6169d3da6e4a71e67a256fea531b78bbad76 WatchSource:0}: Error finding container 252aafc3b37206597fa016aceb9e6169d3da6e4a71e67a256fea531b78bbad76: Status 404 returned error can't find the container with id 252aafc3b37206597fa016aceb9e6169d3da6e4a71e67a256fea531b78bbad76
Jul 12 17:51:18.418243 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:51:18.418205    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rpjth/perf-node-gather-daemonset-hnccj" event={"ID":"be0a1fb9-5e64-4622-abea-d03eae829bea","Type":"ContainerStarted","Data":"ca8cf19f9d50750e5ec83ee5c00f16cccfa135149a22961b3c69e298a50a23f5"}
Jul 12 17:51:18.418415 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:51:18.418254    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rpjth/perf-node-gather-daemonset-hnccj" event={"ID":"be0a1fb9-5e64-4622-abea-d03eae829bea","Type":"ContainerStarted","Data":"252aafc3b37206597fa016aceb9e6169d3da6e4a71e67a256fea531b78bbad76"}
Jul 12 17:51:18.418415 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:51:18.418297    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-rpjth/perf-node-gather-daemonset-hnccj"
Jul 12 17:51:18.432721 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:51:18.432656    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-rpjth/perf-node-gather-daemonset-hnccj" podStartSLOduration=1.432643125 podStartE2EDuration="1.432643125s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-12 17:51:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 17:51:18.43175984 +0000 UTC m=+2886.552076635" watchObservedRunningTime="2026-07-12 17:51:18.432643125 +0000 UTC m=+2886.552959922"
Jul 12 17:51:20.330164 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:51:20.330136    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-2hvcl_e97cfac9-53f7-40e4-97ed-fb725d04971c/dns/0.log"
Jul 12 17:51:20.352374 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:51:20.352349    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-2hvcl_e97cfac9-53f7-40e4-97ed-fb725d04971c/kube-rbac-proxy/0.log"
Jul 12 17:51:20.507315 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:51:20.507287    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-zlfcq_fcdb5e54-4227-4aed-bcb8-b019fc5dfd86/dns-node-resolver/0.log"
Jul 12 17:51:21.065030 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:51:21.064995    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-gpzg5_098853ee-bad5-44d4-b2ac-bc4aa97b5d1c/node-ca/0.log"
Jul 12 17:51:22.092334 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:51:22.092306    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-59584446bf-bxv2x_327a7007-ca31-49a8-acda-91bf35ed6963/kube-auth-proxy/0.log"
Jul 12 17:51:22.988642 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:51:22.988598    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-ttcq7_09c551dc-74e0-4241-9d84-2cd264d717d9/serve-healthcheck-canary/0.log"
Jul 12 17:51:23.507963 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:51:23.507930    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-xztsd_5c7f573f-607f-40cd-b450-1976b2898b83/kube-rbac-proxy/0.log"
Jul 12 17:51:23.527788 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:51:23.527765    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-xztsd_5c7f573f-607f-40cd-b450-1976b2898b83/exporter/0.log"
Jul 12 17:51:23.549142 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:51:23.549123    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-xztsd_5c7f573f-607f-40cd-b450-1976b2898b83/extractor/0.log"
Jul 12 17:51:24.431347 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:51:24.431323    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-rpjth/perf-node-gather-daemonset-hnccj"
Jul 12 17:51:26.282444 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:51:26.282418    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-554bd9d4cc-b2lhx_1aa36e66-d3fd-4083-b35a-e62fefcacdd9/manager/0.log"
Jul 12 17:51:26.304634 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:51:26.304611    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-554bd9d4cc-b2lhx_1aa36e66-d3fd-4083-b35a-e62fefcacdd9/copy-manifests/0.log"
Jul 12 17:51:27.808629 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:51:27.808603    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-6459574568-7rcnn_bccd8821-f075-42d5-88a0-28530ae56bac/manager/0.log"
Jul 12 17:51:28.863454 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:51:28.863384    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-59584446bf-bxv2x_327a7007-ca31-49a8-acda-91bf35ed6963/kube-auth-proxy/0.log"
Jul 12 17:51:30.412928 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:51:30.412901    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-59584446bf-bxv2x_327a7007-ca31-49a8-acda-91bf35ed6963/kube-auth-proxy/0.log"
Jul 12 17:51:34.442776 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:51:34.442750    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ltcld_59055e87-8c03-4225-97e1-62114754af67/kube-multus-additional-cni-plugins/0.log"
Jul 12 17:51:34.462883 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:51:34.462859    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ltcld_59055e87-8c03-4225-97e1-62114754af67/egress-router-binary-copy/0.log"
Jul 12 17:51:34.481814 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:51:34.481788    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ltcld_59055e87-8c03-4225-97e1-62114754af67/cni-plugins/0.log"
Jul 12 17:51:34.500820 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:51:34.500803    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ltcld_59055e87-8c03-4225-97e1-62114754af67/bond-cni-plugin/0.log"
Jul 12 17:51:34.522429 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:51:34.522394    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ltcld_59055e87-8c03-4225-97e1-62114754af67/routeoverride-cni/0.log"
Jul 12 17:51:34.541373 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:51:34.541355    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ltcld_59055e87-8c03-4225-97e1-62114754af67/whereabouts-cni-bincopy/0.log"
Jul 12 17:51:34.562638 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:51:34.562615    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ltcld_59055e87-8c03-4225-97e1-62114754af67/whereabouts-cni/0.log"
Jul 12 17:51:34.734010 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:51:34.733950    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-xsskc_ecab5f05-562e-47bc-9c5a-04893ab8a903/kube-multus/0.log"
Jul 12 17:51:34.864035 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:51:34.863993    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-h442x_f1d6c71b-d9fb-4f2a-9a81-5e8f0f8a6c13/network-metrics-daemon/0.log"
Jul 12 17:51:34.882389 ip-10-0-141-94 kubenswrapper[2569]: I0712 17:51:34.882366    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-h442x_f1d6c71b-d9fb-4f2a-9a81-5e8f0f8a6c13/kube-rbac-proxy/0.log"