Jul 16 00:40:27.473195 ip-10-0-143-65 systemd[1]: Starting Kubernetes Kubelet...
Jul 16 00:40:27.768974 ip-10-0-143-65 kubenswrapper[2563]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 16 00:40:27.768974 ip-10-0-143-65 kubenswrapper[2563]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 16 00:40:27.768974 ip-10-0-143-65 kubenswrapper[2563]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 16 00:40:27.768974 ip-10-0-143-65 kubenswrapper[2563]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 16 00:40:27.768974 ip-10-0-143-65 kubenswrapper[2563]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 16 00:40:27.769633 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.769561    2563 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 16 00:40:27.773253 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.773238    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 16 00:40:27.773253 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.773253    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 16 00:40:27.773319 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.773257    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 16 00:40:27.773319 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.773261    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 16 00:40:27.773319 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.773264    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 16 00:40:27.773319 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.773267    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 16 00:40:27.773319 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.773270    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 16 00:40:27.773319 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.773273    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 16 00:40:27.773319 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.773275    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 16 00:40:27.773319 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.773278    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 16 00:40:27.773319 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.773280    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 16 00:40:27.773319 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.773283    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 16 00:40:27.773319 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.773285    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 16 00:40:27.773319 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.773288    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 16 00:40:27.773319 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.773291    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 16 00:40:27.773319 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.773294    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 16 00:40:27.773319 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.773297    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 16 00:40:27.773319 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.773299    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 16 00:40:27.773319 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.773302    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 16 00:40:27.773319 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.773305    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 16 00:40:27.773319 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.773307    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 16 00:40:27.773319 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.773316    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 16 00:40:27.773802 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.773319    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 16 00:40:27.773802 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.773322    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 16 00:40:27.773802 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.773325    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 16 00:40:27.773802 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.773328    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 16 00:40:27.773802 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.773331    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 16 00:40:27.773802 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.773333    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 16 00:40:27.773802 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.773336    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 16 00:40:27.773802 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.773339    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 16 00:40:27.773802 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.773341    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 16 00:40:27.773802 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.773344    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 16 00:40:27.773802 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.773346    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 16 00:40:27.773802 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.773349    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 16 00:40:27.773802 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.773351    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 16 00:40:27.773802 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.773354    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 16 00:40:27.773802 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.773356    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 16 00:40:27.773802 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.773361    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 16 00:40:27.773802 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.773365    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 16 00:40:27.773802 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.773368    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 16 00:40:27.773802 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.773371    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 16 00:40:27.773802 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.773374    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 16 00:40:27.774270 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.773377    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 16 00:40:27.774270 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.773379    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 16 00:40:27.774270 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.773382    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 16 00:40:27.774270 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.773385    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 16 00:40:27.774270 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.773387    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 16 00:40:27.774270 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.773390    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 16 00:40:27.774270 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.773392    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 16 00:40:27.774270 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.773394    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 16 00:40:27.774270 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.773397    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 16 00:40:27.774270 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.773400    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 16 00:40:27.774270 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.773402    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 16 00:40:27.774270 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.773405    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 16 00:40:27.774270 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.773408    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 16 00:40:27.774270 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.773413    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 16 00:40:27.774270 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.773415    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 16 00:40:27.774270 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.773418    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 16 00:40:27.774270 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.773420    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 16 00:40:27.774270 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.773423    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 16 00:40:27.774270 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.773425    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 16 00:40:27.774270 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.773428    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 16 00:40:27.774757 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.773430    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 16 00:40:27.774757 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.773433    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 16 00:40:27.774757 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.773436    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 16 00:40:27.774757 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.773438    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 16 00:40:27.774757 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.773441    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 16 00:40:27.774757 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.773443    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 16 00:40:27.774757 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.773446    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 16 00:40:27.774757 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.773448    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 16 00:40:27.774757 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.773451    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 16 00:40:27.774757 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.773453    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 16 00:40:27.774757 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.773456    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 16 00:40:27.774757 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.773458    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 16 00:40:27.774757 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.773461    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 16 00:40:27.774757 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.773463    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 16 00:40:27.774757 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.773466    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 16 00:40:27.774757 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.773469    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 16 00:40:27.774757 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.773473    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 16 00:40:27.774757 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.773476    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 16 00:40:27.774757 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.773479    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 16 00:40:27.774757 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.773482    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 16 00:40:27.775237 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.773485    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 16 00:40:27.775237 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.773487    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 16 00:40:27.775237 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.773490    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 16 00:40:27.775237 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.773493    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 16 00:40:27.775237 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.773495    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 16 00:40:27.775237 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.773499    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 16 00:40:27.775237 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.773502    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 16 00:40:27.775237 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.773505    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 16 00:40:27.775237 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.773507    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 16 00:40:27.775237 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.773510    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 16 00:40:27.775237 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.773513    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 16 00:40:27.775237 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.773515    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 16 00:40:27.775237 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.773518    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 16 00:40:27.775237 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.773520    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 16 00:40:27.775237 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.773524    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 16 00:40:27.775237 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.773527    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 16 00:40:27.775237 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.773530    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 16 00:40:27.775237 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.774238    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 16 00:40:27.775237 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.774244    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 16 00:40:27.775714 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.774248    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 16 00:40:27.775714 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.774251    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 16 00:40:27.775714 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.774254    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 16 00:40:27.775714 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.774257    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 16 00:40:27.775714 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.774260    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 16 00:40:27.775714 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.774263    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 16 00:40:27.775714 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.774266    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 16 00:40:27.775714 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.774269    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 16 00:40:27.775714 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.774271    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 16 00:40:27.775714 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.774274    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 16 00:40:27.775714 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.774277    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 16 00:40:27.775714 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.774280    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 16 00:40:27.775714 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.774282    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 16 00:40:27.775714 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.774285    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 16 00:40:27.775714 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.774288    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 16 00:40:27.775714 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.774290    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 16 00:40:27.775714 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.774293    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 16 00:40:27.775714 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.774295    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 16 00:40:27.775714 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.774298    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 16 00:40:27.775714 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.774302    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 16 00:40:27.776216 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.774305    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 16 00:40:27.776216 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.774307    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 16 00:40:27.776216 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.774310    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 16 00:40:27.776216 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.774313    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 16 00:40:27.776216 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.774315    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 16 00:40:27.776216 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.774318    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 16 00:40:27.776216 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.774320    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 16 00:40:27.776216 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.774323    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 16 00:40:27.776216 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.774326    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 16 00:40:27.776216 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.774328    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 16 00:40:27.776216 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.774330    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 16 00:40:27.776216 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.774333    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 16 00:40:27.776216 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.774336    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 16 00:40:27.776216 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.774339    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 16 00:40:27.776216 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.774342    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 16 00:40:27.776216 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.774344    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 16 00:40:27.776216 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.774347    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 16 00:40:27.776216 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.774349    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 16 00:40:27.776216 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.774352    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 16 00:40:27.776703 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.774354    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 16 00:40:27.776703 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.774358    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 16 00:40:27.776703 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.774362    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 16 00:40:27.776703 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.774364    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 16 00:40:27.776703 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.774367    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 16 00:40:27.776703 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.774370    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 16 00:40:27.776703 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.774372    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 16 00:40:27.776703 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.774375    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 16 00:40:27.776703 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.774377    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 16 00:40:27.776703 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.774380    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 16 00:40:27.776703 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.774382    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 16 00:40:27.776703 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.774385    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 16 00:40:27.776703 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.774388    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 16 00:40:27.776703 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.774390    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 16 00:40:27.776703 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.774393    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 16 00:40:27.776703 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.774396    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 16 00:40:27.776703 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.774398    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 16 00:40:27.776703 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.774400    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 16 00:40:27.776703 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.774403    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 16 00:40:27.776703 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.774405    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 16 00:40:27.777200 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.774408    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 16 00:40:27.777200 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.774410    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 16 00:40:27.777200 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.774413    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 16 00:40:27.777200 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.774415    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 16 00:40:27.777200 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.774418    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 16 00:40:27.777200 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.774421    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 16 00:40:27.777200 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.774423    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 16 00:40:27.777200 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.774426    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 16 00:40:27.777200 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.774428    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 16 00:40:27.777200 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.774432    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 16 00:40:27.777200 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.774435    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 16 00:40:27.777200 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.774437    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 16 00:40:27.777200 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.774439    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 16 00:40:27.777200 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.774442    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 16 00:40:27.777200 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.774444    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 16 00:40:27.777200 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.774447    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 16 00:40:27.777200 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.774450    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 16 00:40:27.777200 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.774452    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 16 00:40:27.777200 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.774455    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 16 00:40:27.777200 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.774458    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 16 00:40:27.777686 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.774460    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 16 00:40:27.777686 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.774463    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 16 00:40:27.777686 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.774465    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 16 00:40:27.777686 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.774468    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 16 00:40:27.777686 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.774471    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 16 00:40:27.777686 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.774474    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 16 00:40:27.777686 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.774477    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 16 00:40:27.777686 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.774481    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 16 00:40:27.777686 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.774485    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 16 00:40:27.777686 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.774488    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 16 00:40:27.777686 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.774491    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 16 00:40:27.777686 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.774494    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 16 00:40:27.777686 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.774497    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 16 00:40:27.777686 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.774499    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 16 00:40:27.777686 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.774502    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 16 00:40:27.777686 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.774505    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 16 00:40:27.777686 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.774508    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 16 00:40:27.777686 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.774510    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 16 00:40:27.777686 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.774886    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 16 00:40:27.778152 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.774892    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 16 00:40:27.778152 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.774896    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 16 00:40:27.778152 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.774900    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 16 00:40:27.778152 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.774903    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 16 00:40:27.778152 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.774907    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 16 00:40:27.778152 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.774910    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 16 00:40:27.778152 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.774913    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 16 00:40:27.778152 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.774916    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 16 00:40:27.778152 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.774919    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 16 00:40:27.778152 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.774922    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 16 00:40:27.778152 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.774925    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 16 00:40:27.778152 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.774928    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 16 00:40:27.778152 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.774931    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 16 00:40:27.778152 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.774934    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 16 00:40:27.778152 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.774937    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 16 00:40:27.778152 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.774940    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 16 00:40:27.778152 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.774943    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 16 00:40:27.778152 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.774945    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 16 00:40:27.778152 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.774948    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 16 00:40:27.778606 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.774952    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 16 00:40:27.778606 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.774954    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 16 00:40:27.778606 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.774957    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 16 00:40:27.778606 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.774960    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 16 00:40:27.778606 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.774962    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 16 00:40:27.778606 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.774965    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 16 00:40:27.778606 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.774967    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 16 00:40:27.778606 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.774970    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 16 00:40:27.778606 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.774972    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 16 00:40:27.778606 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.774975    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 16 00:40:27.778606 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.774977    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 16 00:40:27.778606 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.774980    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 16 00:40:27.778606 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.774982    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 16 00:40:27.778606 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.774985    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 16 00:40:27.778606 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.774988    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 16 00:40:27.778606 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.774991    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 16 00:40:27.778606 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.774993    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 16 00:40:27.778606 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.774996    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 16 00:40:27.778606 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.774999    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 16 00:40:27.778606 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.775001    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 16 00:40:27.779088 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.775004    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 16 00:40:27.779088 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.775007    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 16 00:40:27.779088 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.775010    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 16 00:40:27.779088 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.775012    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 16 00:40:27.779088 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.775015    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 16 00:40:27.779088 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.775017    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 16 00:40:27.779088 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.775020    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 16 00:40:27.779088 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.775022    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 16 00:40:27.779088 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.775025    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 16 00:40:27.779088 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.775027    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 16 00:40:27.779088 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.775031    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 16 00:40:27.779088 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.775034    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 16 00:40:27.779088 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.775037    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 16 00:40:27.779088 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.775040    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 16 00:40:27.779088 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.775042    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 16 00:40:27.779088 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.775045    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 16 00:40:27.779088 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.775047    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 16 00:40:27.779088 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.775050    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 16 00:40:27.779088 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.775052    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 16 00:40:27.779088 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.775055    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 16 00:40:27.779563 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.775057    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 16 00:40:27.779563 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.775061    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 16 00:40:27.779563 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.775064    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 16 00:40:27.779563 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.775067    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 16 00:40:27.779563 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.775070    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 16 00:40:27.779563 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.775073    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 16 00:40:27.779563 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.775076    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 16 00:40:27.779563 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.775079    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 16 00:40:27.779563 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.775081    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 16 00:40:27.779563 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.775084    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 16 00:40:27.779563 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.775087    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 16 00:40:27.779563 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.775090    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 16 00:40:27.779563 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.775093    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 16 00:40:27.779563 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.775095    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 16 00:40:27.779563 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.775098    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 16 00:40:27.779563 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.775100    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 16 00:40:27.779563 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.775103    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 16 00:40:27.779563 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.775105    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 16 00:40:27.779563 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.775108    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 16 00:40:27.779563 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.775110    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 16 00:40:27.780076 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.775113    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 16 00:40:27.780076 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.775115    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 16 00:40:27.780076 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.775118    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 16 00:40:27.780076 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.775120    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 16 00:40:27.780076 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.775122    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 16 00:40:27.780076 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.775125    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 16 00:40:27.780076 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.775128    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 16 00:40:27.780076 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.775131    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 16 00:40:27.780076 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.775133    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 16 00:40:27.780076 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.775135    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 16 00:40:27.780076 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.775138    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 16 00:40:27.780076 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.775141    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 16 00:40:27.780076 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.775143    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 16 00:40:27.780076 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.775146    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 16 00:40:27.780076 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.775148    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 16 00:40:27.780076 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.775151    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 16 00:40:27.780076 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.775153    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 16 00:40:27.780076 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.775156    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 16 00:40:27.780076 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.775159    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 16 00:40:27.780076 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.775227    2563 flags.go:64] FLAG: --address="0.0.0.0"
Jul 16 00:40:27.780076 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.775234    2563 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 16 00:40:27.780559 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.775243    2563 flags.go:64] FLAG: --anonymous-auth="true"
Jul 16 00:40:27.780559 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.775248    2563 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 16 00:40:27.780559 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.775252    2563 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 16 00:40:27.780559 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.775255    2563 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 16 00:40:27.780559 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.775259    2563 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 16 00:40:27.780559 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.775263    2563 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 16 00:40:27.780559 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.775266    2563 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 16 00:40:27.780559 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.775269    2563 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 16 00:40:27.780559 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.775273    2563 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 16 00:40:27.780559 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.775276    2563 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 16 00:40:27.780559 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.775279    2563 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 16 00:40:27.780559 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.775281    2563 flags.go:64] FLAG: --cgroup-root=""
Jul 16 00:40:27.780559 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.775285    2563 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 16 00:40:27.780559 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.775288    2563 flags.go:64] FLAG: --client-ca-file=""
Jul 16 00:40:27.780559 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.775291    2563 flags.go:64] FLAG: --cloud-provider="external"
Jul 16 00:40:27.780559 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.775294    2563 flags.go:64] FLAG: --cluster-dns="[]"
Jul 16 00:40:27.780559 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.775298    2563 flags.go:64] FLAG: --cluster-domain=""
Jul 16 00:40:27.780559 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.775301    2563 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 16 00:40:27.780559 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.775311    2563 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 16 00:40:27.780559 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.775314    2563 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 16 00:40:27.780559 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.775318    2563 flags.go:64] FLAG: --container-log-max-files="5"
Jul 16 00:40:27.780559 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.775326    2563 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 16 00:40:27.780559 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.775329    2563 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 16 00:40:27.781116 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.775332    2563 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 16 00:40:27.781116 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.775335    2563 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 16 00:40:27.781116 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.775339    2563 flags.go:64] FLAG: --contention-profiling="false"
Jul 16 00:40:27.781116 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.775341    2563 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 16 00:40:27.781116 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.775344    2563 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 16 00:40:27.781116 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.775349    2563 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 16 00:40:27.781116 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.775352    2563 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 16 00:40:27.781116 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.775360    2563 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 16 00:40:27.781116 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.775363    2563 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 16 00:40:27.781116 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.775368    2563 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 16 00:40:27.781116 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.775371    2563 flags.go:64] FLAG: --enable-load-reader="false"
Jul 16 00:40:27.781116 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.775374    2563 flags.go:64] FLAG: --enable-server="true"
Jul 16 00:40:27.781116 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.775377    2563 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 16 00:40:27.781116 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.775381    2563 flags.go:64] FLAG: --event-burst="100"
Jul 16 00:40:27.781116 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.775384    2563 flags.go:64] FLAG: --event-qps="50"
Jul 16 00:40:27.781116 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.775387    2563 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 16 00:40:27.781116 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.775390    2563 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 16 00:40:27.781116 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.775393    2563 flags.go:64] FLAG: --eviction-hard=""
Jul 16 00:40:27.781116 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.775397    2563 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 16 00:40:27.781116 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.775399    2563 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 16 00:40:27.781116 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.775403    2563 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 16 00:40:27.781116 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.775406    2563 flags.go:64] FLAG: --eviction-soft=""
Jul 16 00:40:27.781116 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.775409    2563 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 16 00:40:27.781116 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.775412    2563 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 16 00:40:27.781116 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.775415    2563 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 16 00:40:27.781766 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.775418    2563 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 16 00:40:27.781766 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.775420    2563 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 16 00:40:27.781766 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.775423    2563 flags.go:64] FLAG: --fail-swap-on="true"
Jul 16 00:40:27.781766 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.775426    2563 flags.go:64] FLAG: --feature-gates=""
Jul 16 00:40:27.781766 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.775430    2563 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 16 00:40:27.781766 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.775433    2563 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 16 00:40:27.781766 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.775437    2563 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 16 00:40:27.781766 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.775439    2563 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 16 00:40:27.781766 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.775442    2563 flags.go:64] FLAG: --healthz-port="10248"
Jul 16 00:40:27.781766 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.775445    2563 flags.go:64] FLAG: --help="false"
Jul 16 00:40:27.781766 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.775448    2563 flags.go:64] FLAG: --hostname-override="ip-10-0-143-65.ec2.internal"
Jul 16 00:40:27.781766 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.775451    2563 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 16 00:40:27.781766 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.775456    2563 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 16 00:40:27.781766 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.775458    2563 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 16 00:40:27.781766 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.775462    2563 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 16 00:40:27.781766 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.775465    2563 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 16 00:40:27.781766 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.775469    2563 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 16 00:40:27.781766 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.775472    2563 flags.go:64] FLAG: --image-service-endpoint=""
Jul 16 00:40:27.781766 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.775475    2563 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 16 00:40:27.781766 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.775478    2563 flags.go:64] FLAG: --kube-api-burst="100"
Jul 16 00:40:27.781766 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.775481    2563 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 16 00:40:27.781766 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.775484    2563 flags.go:64] FLAG: --kube-api-qps="50"
Jul 16 00:40:27.781766 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.775487    2563 flags.go:64] FLAG: --kube-reserved=""
Jul 16 00:40:27.781766 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.775490    2563 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 16 00:40:27.782517 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.775493    2563 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 16 00:40:27.782517 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.775496    2563 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 16 00:40:27.782517 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.775498    2563 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 16 00:40:27.782517 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.775501    2563 flags.go:64] FLAG: --lock-file=""
Jul 16 00:40:27.782517 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.775504    2563 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 16 00:40:27.782517 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.775507    2563 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 16 00:40:27.782517 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.775510    2563 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 16 00:40:27.782517 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.775515    2563 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 16 00:40:27.782517 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.775518    2563 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 16 00:40:27.782517 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.775521    2563 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 16 00:40:27.782517 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.775524    2563 flags.go:64] FLAG: --logging-format="text"
Jul 16 00:40:27.782517 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.775526    2563 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 16 00:40:27.782517 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.775530    2563 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 16 00:40:27.782517 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.775533    2563 flags.go:64] FLAG: --manifest-url=""
Jul 16 00:40:27.782517 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.775536    2563 flags.go:64] FLAG: --manifest-url-header=""
Jul 16 00:40:27.782517 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.775540    2563 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 16 00:40:27.782517 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.775543    2563 flags.go:64] FLAG: --max-open-files="1000000"
Jul 16 00:40:27.782517 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.775547    2563 flags.go:64] FLAG: --max-pods="110"
Jul 16 00:40:27.782517 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.775550    2563 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 16 00:40:27.782517 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.775553    2563 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 16 00:40:27.782517 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.775557    2563 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 16 00:40:27.782517 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.775560    2563 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 16 00:40:27.782517 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.775563    2563 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 16 00:40:27.782517 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.775566    2563 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 16 00:40:27.782517 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.775570    2563 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 16 00:40:27.783395 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.775577    2563 flags.go:64] FLAG: --node-status-max-images="50"
Jul 16 00:40:27.783395 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.775580    2563 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 16 00:40:27.783395 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.775583    2563 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 16 00:40:27.783395 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.775586    2563 flags.go:64] FLAG: --pod-cidr=""
Jul 16 00:40:27.783395 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.775589    2563 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926"
Jul 16 00:40:27.783395 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.775597    2563 flags.go:64] FLAG: --pod-manifest-path=""
Jul 16 00:40:27.783395 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.775600    2563 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 16 00:40:27.783395 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.775603    2563 flags.go:64] FLAG: --pods-per-core="0"
Jul 16 00:40:27.783395 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.775606    2563 flags.go:64] FLAG: --port="10250"
Jul 16 00:40:27.783395 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.775609    2563 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 16 00:40:27.783395 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.775612    2563 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0ff500ef7f2d2ccab"
Jul 16 00:40:27.783395 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.775615    2563 flags.go:64] FLAG: --qos-reserved=""
Jul 16 00:40:27.783395 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.775618    2563 flags.go:64] FLAG: --read-only-port="10255"
Jul 16 00:40:27.783395 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.775621    2563 flags.go:64] FLAG: --register-node="true"
Jul 16 00:40:27.783395 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.775624    2563 flags.go:64] FLAG: --register-with-taints=""
Jul 16 00:40:27.783395 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.775628    2563 flags.go:64] FLAG: --registry-burst="10"
Jul 16 00:40:27.783395 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.775631    2563 flags.go:64] FLAG: --registry-qps="5"
Jul 16 00:40:27.783395 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.775633    2563 flags.go:64] FLAG: --reserved-cpus=""
Jul 16 00:40:27.783395 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.775636    2563 flags.go:64] FLAG: --reserved-memory=""
Jul 16 00:40:27.783395 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.775640    2563 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 16 00:40:27.783395 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.775643    2563 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 16 00:40:27.783395 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.775646    2563 flags.go:64] FLAG: --rotate-certificates="false"
Jul 16 00:40:27.783395 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.775649    2563 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 16 00:40:27.783395 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.775652    2563 flags.go:64] FLAG: --runonce="false"
Jul 16 00:40:27.783395 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.775655    2563 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 16 00:40:27.784030 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.775658    2563 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 16 00:40:27.784030 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.775661    2563 flags.go:64] FLAG: --seccomp-default="false"
Jul 16 00:40:27.784030 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.775666    2563 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 16 00:40:27.784030 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.775669    2563 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 16 00:40:27.784030 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.775672    2563 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 16 00:40:27.784030 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.775675    2563 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 16 00:40:27.784030 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.775677    2563 flags.go:64] FLAG: --storage-driver-password="root"
Jul 16 00:40:27.784030 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.775683    2563 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 16 00:40:27.784030 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.775685    2563 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 16 00:40:27.784030 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.775689    2563 flags.go:64] FLAG: --storage-driver-user="root"
Jul 16 00:40:27.784030 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.775691    2563 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 16 00:40:27.784030 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.775694    2563 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 16 00:40:27.784030 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.775697    2563 flags.go:64] FLAG: --system-cgroups=""
Jul 16 00:40:27.784030 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.775700    2563 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 16 00:40:27.784030 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.775705    2563 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 16 00:40:27.784030 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.775708    2563 flags.go:64] FLAG: --tls-cert-file=""
Jul 16 00:40:27.784030 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.775711    2563 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 16 00:40:27.784030 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.775715    2563 flags.go:64] FLAG: --tls-min-version=""
Jul 16 00:40:27.784030 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.775718    2563 flags.go:64] FLAG: --tls-private-key-file=""
Jul 16 00:40:27.784030 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.775721    2563 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 16 00:40:27.784030 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.775724    2563 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 16 00:40:27.784030 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.775727    2563 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 16 00:40:27.784030 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.775730    2563 flags.go:64] FLAG: --v="2"
Jul 16 00:40:27.784030 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.775734    2563 flags.go:64] FLAG: --version="false"
Jul 16 00:40:27.784030 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.775738    2563 flags.go:64] FLAG: --vmodule=""
Jul 16 00:40:27.784618 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.775741    2563 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 16 00:40:27.784618 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.775745    2563 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 16 00:40:27.784618 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.775857    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 16 00:40:27.784618 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.775880    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 16 00:40:27.784618 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.775884    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 16 00:40:27.784618 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.775887    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 16 00:40:27.784618 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.775890    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 16 00:40:27.784618 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.775893    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 16 00:40:27.784618 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.775896    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 16 00:40:27.784618 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.775901    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 16 00:40:27.784618 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.775904    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 16 00:40:27.784618 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.775907    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 16 00:40:27.784618 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.775909    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 16 00:40:27.784618 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.775912    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 16 00:40:27.784618 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.775915    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 16 00:40:27.784618 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.775918    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 16 00:40:27.784618 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.775921    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 16 00:40:27.784618 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.775924    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 16 00:40:27.784618 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.775926    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 16 00:40:27.784618 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.775929    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 16 00:40:27.785132 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.775931    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 16 00:40:27.785132 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.775934    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 16 00:40:27.785132 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.775937    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 16 00:40:27.785132 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.775939    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 16 00:40:27.785132 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.775942    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 16 00:40:27.785132 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.775944    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 16 00:40:27.785132 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.775947    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 16 00:40:27.785132 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.775950    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 16 00:40:27.785132 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.775952    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 16 00:40:27.785132 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.775955    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 16 00:40:27.785132 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.775957    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 16 00:40:27.785132 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.775960    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 16 00:40:27.785132 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.775963    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 16 00:40:27.785132 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.775965    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 16 00:40:27.785132 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.775967    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 16 00:40:27.785132 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.775970    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 16 00:40:27.785132 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.775973    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 16 00:40:27.785132 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.775976    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 16 00:40:27.785132 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.775978    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 16 00:40:27.785132 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.775981    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 16 00:40:27.785639 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.775984    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 16 00:40:27.785639 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.775988    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 16 00:40:27.785639 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.775991    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 16 00:40:27.785639 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.775993    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 16 00:40:27.785639 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.775996    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 16 00:40:27.785639 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.775998    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 16 00:40:27.785639 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.776002    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 16 00:40:27.785639 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.776005    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 16 00:40:27.785639 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.776008    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 16 00:40:27.785639 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.776011    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 16 00:40:27.785639 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.776013    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 16 00:40:27.785639 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.776016    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 16 00:40:27.785639 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.776018    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 16 00:40:27.785639 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.776021    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 16 00:40:27.785639 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.776023    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 16 00:40:27.785639 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.776026    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 16 00:40:27.785639 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.776029    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 16 00:40:27.785639 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.776031    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 16 00:40:27.785639 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.776034    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 16 00:40:27.785639 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.776036    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 16 00:40:27.786264 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.776039    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 16 00:40:27.786264 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.776041    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 16 00:40:27.786264 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.776044    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 16 00:40:27.786264 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.776047    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 16 00:40:27.786264 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.776049    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 16 00:40:27.786264 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.776052    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 16 00:40:27.786264 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.776055    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 16 00:40:27.786264 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.776057    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 16 00:40:27.786264 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.776060    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 16 00:40:27.786264 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.776063    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 16 00:40:27.786264 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.776066    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 16 00:40:27.786264 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.776068    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 16 00:40:27.786264 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.776072    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 16 00:40:27.786264 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.776077    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 16 00:40:27.786264 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.776080    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 16 00:40:27.786264 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.776083    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 16 00:40:27.786264 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.776086    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 16 00:40:27.786264 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.776088    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 16 00:40:27.786264 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.776092    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 16 00:40:27.786708 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.776095    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 16 00:40:27.786708 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.776098    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 16 00:40:27.786708 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.776100    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 16 00:40:27.786708 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.776103    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 16 00:40:27.786708 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.776105    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 16 00:40:27.786708 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.776108    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 16 00:40:27.786708 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.776110    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 16 00:40:27.786708 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.776114    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 16 00:40:27.786708 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.776118    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 16 00:40:27.786708 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.776121    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 16 00:40:27.786708 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.776124    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 16 00:40:27.786708 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.776127    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 16 00:40:27.786708 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.776129    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 16 00:40:27.786708 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.776133    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 16 00:40:27.786708 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.776136    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 16 00:40:27.786708 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.776140    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 16 00:40:27.786708 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.776142    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 16 00:40:27.786708 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.776145    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 16 00:40:27.786708 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.776148    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 16 00:40:27.787186 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.776150    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 16 00:40:27.787186 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.776153    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 16 00:40:27.787186 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.776156    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 16 00:40:27.787186 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.776161    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 16 00:40:27.787186 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.783813    2563 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 16 00:40:27.787186 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.783835    2563 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 16 00:40:27.787186 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.783855    2563 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 16 00:40:27.787186 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.783877    2563 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 16 00:40:27.787186 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.783940    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 16 00:40:27.787186 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.783946    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 16 00:40:27.787186 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.783950    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 16 00:40:27.787186 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.783953    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 16 00:40:27.787186 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.783958    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 16 00:40:27.787186 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.783962    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 16 00:40:27.787530 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.783965    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 16 00:40:27.787530 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.783968    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 16 00:40:27.787530 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.783971    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 16 00:40:27.787530 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.783975    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 16 00:40:27.787530 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.783978    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 16 00:40:27.787530 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.783981    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 16 00:40:27.787530 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.783983    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 16 00:40:27.787530 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.783988    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 16 00:40:27.787530 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.783991    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 16 00:40:27.787530 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.783993    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 16 00:40:27.787530 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.783996    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 16 00:40:27.787530 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.783998    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 16 00:40:27.787530 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784001    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 16 00:40:27.787530 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784003    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 16 00:40:27.787530 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784006    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 16 00:40:27.787530 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784009    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 16 00:40:27.787530 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784011    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 16 00:40:27.787530 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784015    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 16 00:40:27.787530 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784019    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 16 00:40:27.788000 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784023    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 16 00:40:27.788000 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784027    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 16 00:40:27.788000 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784030    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 16 00:40:27.788000 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784034    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 16 00:40:27.788000 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784037    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 16 00:40:27.788000 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784040    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 16 00:40:27.788000 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784043    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 16 00:40:27.788000 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784045    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 16 00:40:27.788000 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784048    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 16 00:40:27.788000 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784051    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 16 00:40:27.788000 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784053    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 16 00:40:27.788000 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784056    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 16 00:40:27.788000 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784058    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 16 00:40:27.788000 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784061    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 16 00:40:27.788000 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784063    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 16 00:40:27.788000 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784066    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 16 00:40:27.788000 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784069    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 16 00:40:27.788000 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784071    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 16 00:40:27.788000 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784074    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 16 00:40:27.788000 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784077    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 16 00:40:27.788480 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784080    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 16 00:40:27.788480 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784083    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 16 00:40:27.788480 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784085    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 16 00:40:27.788480 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784088    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 16 00:40:27.788480 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784090    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 16 00:40:27.788480 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784093    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 16 00:40:27.788480 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784095    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 16 00:40:27.788480 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784098    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 16 00:40:27.788480 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784101    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 16 00:40:27.788480 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784103    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 16 00:40:27.788480 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784106    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 16 00:40:27.788480 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784109    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 16 00:40:27.788480 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784111    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 16 00:40:27.788480 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784114    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 16 00:40:27.788480 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784116    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 16 00:40:27.788480 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784119    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 16 00:40:27.788480 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784122    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 16 00:40:27.788480 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784125    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 16 00:40:27.788480 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784128    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 16 00:40:27.788480 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784130    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 16 00:40:27.789151 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784133    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 16 00:40:27.789151 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784135    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 16 00:40:27.789151 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784138    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 16 00:40:27.789151 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784140    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 16 00:40:27.789151 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784143    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 16 00:40:27.789151 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784145    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 16 00:40:27.789151 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784148    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 16 00:40:27.789151 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784150    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 16 00:40:27.789151 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784153    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 16 00:40:27.789151 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784155    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 16 00:40:27.789151 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784158    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 16 00:40:27.789151 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784161    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 16 00:40:27.789151 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784163    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 16 00:40:27.789151 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784166    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 16 00:40:27.789151 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784169    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 16 00:40:27.789151 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784171    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 16 00:40:27.789151 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784174    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 16 00:40:27.789151 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784176    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 16 00:40:27.789151 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784180    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 16 00:40:27.789620 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784182    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 16 00:40:27.789620 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784185    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 16 00:40:27.789620 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784187    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 16 00:40:27.789620 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784190    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 16 00:40:27.789620 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784192    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 16 00:40:27.789620 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784195    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 16 00:40:27.789620 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784198    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 16 00:40:27.789620 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784200    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 16 00:40:27.789620 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784203    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 16 00:40:27.789620 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784206    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 16 00:40:27.789620 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784209    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 16 00:40:27.789620 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784211    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 16 00:40:27.789620 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784214    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 16 00:40:27.789620 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784217    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 16 00:40:27.789620 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784219    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 16 00:40:27.790016 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.784225    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 16 00:40:27.790016 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784356    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 16 00:40:27.790016 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784362    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 16 00:40:27.790016 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784365    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 16 00:40:27.790016 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784368    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 16 00:40:27.790016 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784372    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 16 00:40:27.790016 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784376    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 16 00:40:27.790016 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784379    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 16 00:40:27.790016 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784382    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 16 00:40:27.790016 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784385    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 16 00:40:27.790016 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784388    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 16 00:40:27.790016 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784391    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 16 00:40:27.790016 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784394    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 16 00:40:27.790016 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784397    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 16 00:40:27.790016 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784400    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 16 00:40:27.790383 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784403    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 16 00:40:27.790383 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784406    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 16 00:40:27.790383 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784408    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 16 00:40:27.790383 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784411    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 16 00:40:27.790383 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784414    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 16 00:40:27.790383 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784416    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 16 00:40:27.790383 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784419    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 16 00:40:27.790383 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784422    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 16 00:40:27.790383 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784425    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 16 00:40:27.790383 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784427    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 16 00:40:27.790383 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784430    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 16 00:40:27.790383 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784432    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 16 00:40:27.790383 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784435    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 16 00:40:27.790383 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784438    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 16 00:40:27.790383 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784440    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 16 00:40:27.790383 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784443    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 16 00:40:27.790383 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784445    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 16 00:40:27.790383 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784448    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 16 00:40:27.790383 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784451    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 16 00:40:27.790383 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784453    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 16 00:40:27.790856 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784456    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 16 00:40:27.790856 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784459    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 16 00:40:27.790856 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784461    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 16 00:40:27.790856 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784463    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 16 00:40:27.790856 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784466    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 16 00:40:27.790856 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784468    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 16 00:40:27.790856 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784471    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 16 00:40:27.790856 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784473    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 16 00:40:27.790856 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784476    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 16 00:40:27.790856 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784478    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 16 00:40:27.790856 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784481    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 16 00:40:27.790856 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784484    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 16 00:40:27.790856 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784487    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 16 00:40:27.790856 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784489    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 16 00:40:27.790856 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784491    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 16 00:40:27.790856 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784494    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 16 00:40:27.790856 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784496    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 16 00:40:27.790856 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784499    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 16 00:40:27.790856 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784502    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 16 00:40:27.791351 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784504    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 16 00:40:27.791351 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784508    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 16 00:40:27.791351 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784511    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 16 00:40:27.791351 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784514    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 16 00:40:27.791351 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784517    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 16 00:40:27.791351 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784519    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 16 00:40:27.791351 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784522    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 16 00:40:27.791351 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784524    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 16 00:40:27.791351 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784528    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 16 00:40:27.791351 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784531    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 16 00:40:27.791351 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784535    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 16 00:40:27.791351 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784537    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 16 00:40:27.791351 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784540    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 16 00:40:27.791351 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784543    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 16 00:40:27.791351 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784546    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 16 00:40:27.791351 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784548    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 16 00:40:27.791351 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784551    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 16 00:40:27.791351 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784553    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 16 00:40:27.791351 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784556    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 16 00:40:27.791351 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784558    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 16 00:40:27.791829 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784561    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 16 00:40:27.791829 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784563    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 16 00:40:27.791829 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784566    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 16 00:40:27.791829 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784568    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 16 00:40:27.791829 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784571    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 16 00:40:27.791829 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784573    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 16 00:40:27.791829 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784576    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 16 00:40:27.791829 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784578    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 16 00:40:27.791829 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784581    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 16 00:40:27.791829 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784583    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 16 00:40:27.791829 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784586    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 16 00:40:27.791829 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784588    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 16 00:40:27.791829 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784591    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 16 00:40:27.791829 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784593    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 16 00:40:27.791829 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784596    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 16 00:40:27.791829 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784598    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 16 00:40:27.791829 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784601    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 16 00:40:27.791829 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784604    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 16 00:40:27.791829 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784606    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 16 00:40:27.791829 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784609    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 16 00:40:27.792341 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784611    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 16 00:40:27.792341 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784614    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 16 00:40:27.792341 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784616    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 16 00:40:27.792341 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784619    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 16 00:40:27.792341 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784622    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 16 00:40:27.792341 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:27.784625    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 16 00:40:27.792341 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.784629    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 16 00:40:27.792341 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.785837    2563 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 16 00:40:27.792341 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.788316    2563 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 16 00:40:27.792341 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.789344    2563 server.go:1019] "Starting client certificate rotation"
Jul 16 00:40:27.792341 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.789446    2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 16 00:40:27.792341 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.789480    2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 16 00:40:27.792341 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.791388    2563 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 16 00:40:27.792658 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.792589    2563 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 16 00:40:27.801356 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.801340    2563 log.go:25] "Validated CRI v1 runtime API"
Jul 16 00:40:27.804470 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.804449    2563 log.go:25] "Validated CRI v1 image API"
Jul 16 00:40:27.805838 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.805822    2563 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 16 00:40:27.807729 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.807705    2563 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 d85dde5c-ef88-491b-ab9c-022b9523bc81:/dev/nvme0n1p3 e45a8b79-3a07-46c9-8e47-298d01b76ade:/dev/nvme0n1p4]
Jul 16 00:40:27.807789 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.807729    2563 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 16 00:40:27.812498 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.812368    2563 manager.go:216] Machine: {Timestamp:2026-07-16 00:40:27.811852938 +0000 UTC m=+0.247114669 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101155 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec276d6948e1251c1082309ba5de7382 SystemUUID:ec276d69-48e1-251c-1082-309ba5de7382 BootID:cdbd4044-be90-4751-9d99-e7cdf685a834 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:1c:84:88:63:95 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:1c:84:88:63:95 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:7a:f8:d9:00:ab:f1 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 16 00:40:27.812498 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.812494    2563 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 16 00:40:27.812605 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.812566    2563 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 16 00:40:27.814341 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.814317    2563 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 16 00:40:27.814468 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.814344    2563 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-65.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 16 00:40:27.815331 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.815320    2563 topology_manager.go:138] "Creating topology manager with none policy"
Jul 16 00:40:27.815362 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.815333    2563 container_manager_linux.go:309] "Creating device plugin manager"
Jul 16 00:40:27.815362 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.815341    2563 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 16 00:40:27.815874 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.815852    2563 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 16 00:40:27.816957 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.816946    2563 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 16 00:40:27.818473 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.818459    2563 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 16 00:40:27.820441 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.820430    2563 state_mem.go:36] "Initialized new in-memory state store"
Jul 16 00:40:27.820551 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.820541    2563 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 16 00:40:27.820580 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.820572    2563 kubelet.go:486] "Attempting to sync node with API server"
Jul 16 00:40:27.820613 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.820582    2563 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 16 00:40:27.820613 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.820596    2563 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 16 00:40:27.820613 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.820605    2563 kubelet.go:398] "Adding apiserver pod source"
Jul 16 00:40:27.820613 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.820613    2563 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 16 00:40:27.821039 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.821021    2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 16 00:40:27.821132 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.821053    2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 16 00:40:27.823811 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.823773    2563 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 16 00:40:27.825738 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.825720    2563 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 16 00:40:27.825813 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.825752    2563 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 16 00:40:27.825957 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.825946    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 16 00:40:27.825991 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.825965    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 16 00:40:27.825991 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.825974    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 16 00:40:27.825991 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.825980    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 16 00:40:27.825991 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.825989    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 16 00:40:27.826092 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.825995    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 16 00:40:27.826327 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.826319    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 16 00:40:27.826355 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.826327    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 16 00:40:27.826355 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.826334    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 16 00:40:27.826355 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.826339    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 16 00:40:27.826355 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.826347    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 16 00:40:27.826355 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.826355    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 16 00:40:27.827129 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.827120    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 16 00:40:27.827129 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.827129    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 16 00:40:27.830464 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.830449    2563 server.go:1268] "Started kubelet"
Jul 16 00:40:27.830579 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.830528    2563 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 16 00:40:27.830672 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.830595    2563 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 16 00:40:27.830848 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.830830    2563 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-65.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 16 00:40:27.831013 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:40:27.830984    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 16 00:40:27.831130 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.831097    2563 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 16 00:40:27.831130 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:40:27.831120    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-65.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 16 00:40:27.831345 ip-10-0-143-65 systemd[1]: Started Kubernetes Kubelet.
Jul 16 00:40:27.831911 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.831737    2563 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 16 00:40:27.833253 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.833239    2563 server.go:310] "Adding debug handlers to kubelet server"
Jul 16 00:40:27.835497 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:40:27.835482    2563 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 16 00:40:27.836903 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.836885    2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 16 00:40:27.837349 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.837334    2563 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 16 00:40:27.837488 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:40:27.837446    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-65.ec2.internal\" not found"
Jul 16 00:40:27.837962 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.837943    2563 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 16 00:40:27.838036 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.838004    2563 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 16 00:40:27.838036 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.837955    2563 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 16 00:40:27.838130 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.838090    2563 reconstruct.go:97] "Volume reconstruction finished"
Jul 16 00:40:27.838130 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.838097    2563 reconciler.go:29] "Reconciler: start to sync state"
Jul 16 00:40:27.838210 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.838138    2563 factory.go:55] Registering systemd factory
Jul 16 00:40:27.838210 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.838154    2563 factory.go:223] Registration of the systemd container factory successfully
Jul 16 00:40:27.838374 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.838358    2563 factory.go:153] Registering CRI-O factory
Jul 16 00:40:27.838421 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.838378    2563 factory.go:223] Registration of the crio container factory successfully
Jul 16 00:40:27.838464 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.838425    2563 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 16 00:40:27.838464 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.838444    2563 factory.go:103] Registering Raw factory
Jul 16 00:40:27.838464 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.838454    2563 manager.go:1196] Started watching for new ooms in manager
Jul 16 00:40:27.838798 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:40:27.837885    2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-65.ec2.internal.18c29e46746e557f  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-65.ec2.internal,UID:ip-10-0-143-65.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-143-65.ec2.internal,},FirstTimestamp:2026-07-16 00:40:27.830424959 +0000 UTC m=+0.265686689,LastTimestamp:2026-07-16 00:40:27.830424959 +0000 UTC m=+0.265686689,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-65.ec2.internal,}"
Jul 16 00:40:27.838891 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.838838    2563 manager.go:349] Starting recovery of all containers
Jul 16 00:40:27.842879 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:40:27.842834    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 16 00:40:27.843067 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:40:27.843042    2563 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-65.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 16 00:40:27.848745 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.847311    2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-m8r82"
Jul 16 00:40:27.850388 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.850370    2563 manager.go:354] Recovery completed
Jul 16 00:40:27.854938 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.854926    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 00:40:27.856757 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.856742    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-65.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 00:40:27.856844 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.856773    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-65.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 00:40:27.856844 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.856788    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-65.ec2.internal" event="NodeHasSufficientPID"
Jul 16 00:40:27.857313 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.857301    2563 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 16 00:40:27.857313 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.857312    2563 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 16 00:40:27.857392 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.857327    2563 state_mem.go:36] "Initialized new in-memory state store"
Jul 16 00:40:27.857392 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.857360    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 00:40:27.858002 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.857986    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-65.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 00:40:27.858074 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.858015    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-65.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 00:40:27.858074 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.858029    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-65.ec2.internal" event="NodeHasSufficientPID"
Jul 16 00:40:27.858646 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:40:27.858582    2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-65.ec2.internal.18c29e4676002227  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-65.ec2.internal,UID:ip-10-0-143-65.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-143-65.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-143-65.ec2.internal,},FirstTimestamp:2026-07-16 00:40:27.856757287 +0000 UTC m=+0.292019024,LastTimestamp:2026-07-16 00:40:27.856757287 +0000 UTC m=+0.292019024,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-65.ec2.internal,}"
Jul 16 00:40:27.859478 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.859464    2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-m8r82"
Jul 16 00:40:27.859574 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.859557    2563 policy_none.go:49] "None policy: Start"
Jul 16 00:40:27.859627 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.859576    2563 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 16 00:40:27.859627 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.859591    2563 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 16 00:40:27.861588 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.861573    2563 policy_none.go:47] "Start"
Jul 16 00:40:27.898465 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.898448    2563 manager.go:337] "Starting Device Plugin manager"
Jul 16 00:40:27.898575 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:40:27.898481    2563 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 16 00:40:27.898575 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.898491    2563 server.go:87] "Starting device plugin registration server"
Jul 16 00:40:27.898740 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.898729    2563 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 16 00:40:27.898785 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.898742    2563 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 16 00:40:27.898837 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.898815    2563 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 16 00:40:27.898969 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.898909    2563 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 16 00:40:27.898969 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.898919    2563 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 16 00:40:27.899494 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:40:27.899473    2563 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 16 00:40:27.899568 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:40:27.899527    2563 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-65.ec2.internal\" not found"
Jul 16 00:40:27.998540 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.998486    2563 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 16 00:40:28.000894 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.998884    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 00:40:28.000894 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.999650    2563 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 16 00:40:28.000894 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.999686    2563 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 16 00:40:28.000894 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:27.999713    2563 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 16 00:40:28.000894 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:40:27.999758    2563 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 16 00:40:28.000894 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:28.000606    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-65.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 00:40:28.000894 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:28.000645    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-65.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 00:40:28.000894 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:28.000688    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-65.ec2.internal" event="NodeHasSufficientPID"
Jul 16 00:40:28.000894 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:28.000721    2563 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-65.ec2.internal"
Jul 16 00:40:28.002401 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:28.002382    2563 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 16 00:40:28.008903 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:28.008890    2563 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-143-65.ec2.internal"
Jul 16 00:40:28.008966 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:40:28.008910    2563 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-143-65.ec2.internal\": node \"ip-10-0-143-65.ec2.internal\" not found"
Jul 16 00:40:28.036757 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:40:28.036696    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-65.ec2.internal\" not found"
Jul 16 00:40:28.100006 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:28.099972    2563 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-65.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-65.ec2.internal"]
Jul 16 00:40:28.100152 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:28.100050    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 00:40:28.101485 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:28.101469    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-65.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 00:40:28.101573 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:28.101499    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-65.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 00:40:28.101573 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:28.101513    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-65.ec2.internal" event="NodeHasSufficientPID"
Jul 16 00:40:28.103928 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:28.103915    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 00:40:28.104032 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:28.104018    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-65.ec2.internal"
Jul 16 00:40:28.104068 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:28.104057    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 00:40:28.104576 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:28.104564    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-65.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 00:40:28.104576 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:28.104571    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-65.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 00:40:28.104670 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:28.104593    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-65.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 00:40:28.104670 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:28.104611    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-65.ec2.internal" event="NodeHasSufficientPID"
Jul 16 00:40:28.104670 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:28.104666    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-65.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 00:40:28.104809 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:28.104684    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-65.ec2.internal" event="NodeHasSufficientPID"
Jul 16 00:40:28.106986 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:28.106969    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-65.ec2.internal"
Jul 16 00:40:28.107079 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:28.106993    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 00:40:28.107663 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:28.107646    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-65.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 00:40:28.107736 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:28.107673    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-65.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 00:40:28.107736 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:28.107684    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-65.ec2.internal" event="NodeHasSufficientPID"
Jul 16 00:40:28.129484 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:40:28.129458    2563 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-65.ec2.internal\" not found" node="ip-10-0-143-65.ec2.internal"
Jul 16 00:40:28.133729 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:40:28.133710    2563 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-65.ec2.internal\" not found" node="ip-10-0-143-65.ec2.internal"
Jul 16 00:40:28.136966 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:40:28.136952    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-65.ec2.internal\" not found"
Jul 16 00:40:28.140218 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:28.140205    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/69184a0cad720aed3f473960e4918c4c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-65.ec2.internal\" (UID: \"69184a0cad720aed3f473960e4918c4c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-65.ec2.internal"
Jul 16 00:40:28.140270 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:28.140225    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/69184a0cad720aed3f473960e4918c4c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-65.ec2.internal\" (UID: \"69184a0cad720aed3f473960e4918c4c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-65.ec2.internal"
Jul 16 00:40:28.140270 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:28.140243    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ebeafacc0368100d06658fd0ec2ae27a-config\") pod \"kube-apiserver-proxy-ip-10-0-143-65.ec2.internal\" (UID: \"ebeafacc0368100d06658fd0ec2ae27a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-65.ec2.internal"
Jul 16 00:40:28.238045 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:40:28.238013    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-65.ec2.internal\" not found"
Jul 16 00:40:28.241390 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:28.241375    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/69184a0cad720aed3f473960e4918c4c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-65.ec2.internal\" (UID: \"69184a0cad720aed3f473960e4918c4c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-65.ec2.internal"
Jul 16 00:40:28.241455 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:28.241399    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/69184a0cad720aed3f473960e4918c4c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-65.ec2.internal\" (UID: \"69184a0cad720aed3f473960e4918c4c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-65.ec2.internal"
Jul 16 00:40:28.241455 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:28.241417    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ebeafacc0368100d06658fd0ec2ae27a-config\") pod \"kube-apiserver-proxy-ip-10-0-143-65.ec2.internal\" (UID: \"ebeafacc0368100d06658fd0ec2ae27a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-65.ec2.internal"
Jul 16 00:40:28.241533 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:28.241444    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ebeafacc0368100d06658fd0ec2ae27a-config\") pod \"kube-apiserver-proxy-ip-10-0-143-65.ec2.internal\" (UID: \"ebeafacc0368100d06658fd0ec2ae27a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-65.ec2.internal"
Jul 16 00:40:28.241533 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:28.241481    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/69184a0cad720aed3f473960e4918c4c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-65.ec2.internal\" (UID: \"69184a0cad720aed3f473960e4918c4c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-65.ec2.internal"
Jul 16 00:40:28.241533 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:28.241493    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/69184a0cad720aed3f473960e4918c4c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-65.ec2.internal\" (UID: \"69184a0cad720aed3f473960e4918c4c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-65.ec2.internal"
Jul 16 00:40:28.338409 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:40:28.338297    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-65.ec2.internal\" not found"
Jul 16 00:40:28.434444 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:28.434409    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-65.ec2.internal"
Jul 16 00:40:28.436753 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:28.436724    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-65.ec2.internal"
Jul 16 00:40:28.438787 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:40:28.438771    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-65.ec2.internal\" not found"
Jul 16 00:40:28.540377 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:40:28.540329    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-65.ec2.internal\" not found"
Jul 16 00:40:28.640900 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:40:28.640811    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-65.ec2.internal\" not found"
Jul 16 00:40:28.742040 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:40:28.742002    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-65.ec2.internal\" not found"
Jul 16 00:40:28.789814 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:28.789779    2563 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 16 00:40:28.790414 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:28.789953    2563 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 16 00:40:28.842336 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:40:28.842304    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-65.ec2.internal\" not found"
Jul 16 00:40:28.863582 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:28.863547    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-15 00:35:28 +0000 UTC" deadline="2028-03-03 03:56:31.72630858 +0000 UTC"
Jul 16 00:40:28.863582 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:28.863577    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14307h16m2.86273521s"
Jul 16 00:40:28.942570 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:40:28.942546    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-65.ec2.internal\" not found"
Jul 16 00:40:29.043239 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:40:29.043204    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-65.ec2.internal\" not found"
Jul 16 00:40:29.059415 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:29.059379    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebeafacc0368100d06658fd0ec2ae27a.slice/crio-4f179a00f926e3186e2b2244e4433625c3c802f3048d5e7cd05caec34ccf4fbd WatchSource:0}: Error finding container 4f179a00f926e3186e2b2244e4433625c3c802f3048d5e7cd05caec34ccf4fbd: Status 404 returned error can't find the container with id 4f179a00f926e3186e2b2244e4433625c3c802f3048d5e7cd05caec34ccf4fbd
Jul 16 00:40:29.065708 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.065693    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 16 00:40:29.071353 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:29.071331    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69184a0cad720aed3f473960e4918c4c.slice/crio-a43ad39d145e33df585a14c4239d0c7bff30e91135d8e35675f1ac5e205d7a2d WatchSource:0}: Error finding container a43ad39d145e33df585a14c4239d0c7bff30e91135d8e35675f1ac5e205d7a2d: Status 404 returned error can't find the container with id a43ad39d145e33df585a14c4239d0c7bff30e91135d8e35675f1ac5e205d7a2d
Jul 16 00:40:29.101201 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.101167    2563 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 16 00:40:29.144035 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:40:29.144007    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-65.ec2.internal\" not found"
Jul 16 00:40:29.183936 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.183906    2563 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 16 00:40:29.245112 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:40:29.245032    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-65.ec2.internal\" not found"
Jul 16 00:40:29.329783 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.328819    2563 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 16 00:40:29.337657 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.337630    2563 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-65.ec2.internal"
Jul 16 00:40:29.373992 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.373967    2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 16 00:40:29.374847 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.374834    2563 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-65.ec2.internal"
Jul 16 00:40:29.394019 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.393999    2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 16 00:40:29.808218 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.808183    2563 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 16 00:40:29.824293 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.824267    2563 apiserver.go:52] "Watching apiserver"
Jul 16 00:40:29.832644 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.832616    2563 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 16 00:40:29.833903 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.833869    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-tm5f5","kube-system/kube-apiserver-proxy-ip-10-0-143-65.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sfpw7","openshift-cluster-node-tuning-operator/tuned-rjk6r","openshift-dns/node-resolver-gzbcl","openshift-image-registry/node-ca-rk85k","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-65.ec2.internal","openshift-multus/multus-additional-cni-plugins-l97c7","kube-system/global-pull-secret-syncer-hgcmx","openshift-multus/multus-h4ks8","openshift-multus/network-metrics-daemon-bkf2h","openshift-network-diagnostics/network-check-target-btw92","openshift-network-operator/iptables-alerter-ksj4j","openshift-ovn-kubernetes/ovnkube-node-bn5hr"]
Jul 16 00:40:29.836722 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.836699    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-h4ks8"
Jul 16 00:40:29.837772 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.837752    2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 16 00:40:29.839984 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.839962    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-ksj4j"
Jul 16 00:40:29.842481 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.842461    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sfpw7"
Jul 16 00:40:29.844796 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.844765    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rjk6r"
Jul 16 00:40:29.844973 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.844844    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 16 00:40:29.845480 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.844772    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 16 00:40:29.845480 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.845145    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 16 00:40:29.847078 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.846886    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 16 00:40:29.847152 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.847121    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 16 00:40:29.847389 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.847373    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-5trw6\""
Jul 16 00:40:29.847638 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.847618    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 16 00:40:29.847718 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.847668    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-grtzs\""
Jul 16 00:40:29.847718 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.847699    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 16 00:40:29.847827 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.847626    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-t4tft\""
Jul 16 00:40:29.847913 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.847813    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 16 00:40:29.848038 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.848020    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-gzbcl"
Jul 16 00:40:29.848676 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.848654    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 16 00:40:29.849161 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.849140    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 16 00:40:29.849382 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.849140    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-xk8pw\""
Jul 16 00:40:29.849591 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.849574    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-rk85k"
Jul 16 00:40:29.850175 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.850151    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/090eddb9-3f4c-4a94-9b62-4f6bf8f38428-multus-socket-dir-parent\") pod \"multus-h4ks8\" (UID: \"090eddb9-3f4c-4a94-9b62-4f6bf8f38428\") " pod="openshift-multus/multus-h4ks8"
Jul 16 00:40:29.850273 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.850189    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/090eddb9-3f4c-4a94-9b62-4f6bf8f38428-host-run-netns\") pod \"multus-h4ks8\" (UID: \"090eddb9-3f4c-4a94-9b62-4f6bf8f38428\") " pod="openshift-multus/multus-h4ks8"
Jul 16 00:40:29.850273 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.850224    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/090eddb9-3f4c-4a94-9b62-4f6bf8f38428-host-var-lib-cni-multus\") pod \"multus-h4ks8\" (UID: \"090eddb9-3f4c-4a94-9b62-4f6bf8f38428\") " pod="openshift-multus/multus-h4ks8"
Jul 16 00:40:29.850273 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.850252    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/090eddb9-3f4c-4a94-9b62-4f6bf8f38428-host-var-lib-kubelet\") pod \"multus-h4ks8\" (UID: \"090eddb9-3f4c-4a94-9b62-4f6bf8f38428\") " pod="openshift-multus/multus-h4ks8"
Jul 16 00:40:29.850430 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.850280    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/090eddb9-3f4c-4a94-9b62-4f6bf8f38428-multus-cni-dir\") pod \"multus-h4ks8\" (UID: \"090eddb9-3f4c-4a94-9b62-4f6bf8f38428\") " pod="openshift-multus/multus-h4ks8"
Jul 16 00:40:29.850430 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.850303    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/090eddb9-3f4c-4a94-9b62-4f6bf8f38428-cnibin\") pod \"multus-h4ks8\" (UID: \"090eddb9-3f4c-4a94-9b62-4f6bf8f38428\") " pod="openshift-multus/multus-h4ks8"
Jul 16 00:40:29.850430 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.850322    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/090eddb9-3f4c-4a94-9b62-4f6bf8f38428-os-release\") pod \"multus-h4ks8\" (UID: \"090eddb9-3f4c-4a94-9b62-4f6bf8f38428\") " pod="openshift-multus/multus-h4ks8"
Jul 16 00:40:29.850430 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.850337    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/090eddb9-3f4c-4a94-9b62-4f6bf8f38428-hostroot\") pod \"multus-h4ks8\" (UID: \"090eddb9-3f4c-4a94-9b62-4f6bf8f38428\") " pod="openshift-multus/multus-h4ks8"
Jul 16 00:40:29.850430 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.850352    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mz4pp\" (UniqueName: \"kubernetes.io/projected/090eddb9-3f4c-4a94-9b62-4f6bf8f38428-kube-api-access-mz4pp\") pod \"multus-h4ks8\" (UID: \"090eddb9-3f4c-4a94-9b62-4f6bf8f38428\") " pod="openshift-multus/multus-h4ks8"
Jul 16 00:40:29.850430 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.850367    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlpjz\" (UniqueName: \"kubernetes.io/projected/ed5df646-0604-408b-bd6e-a502f0bad8e1-kube-api-access-qlpjz\") pod \"iptables-alerter-ksj4j\" (UID: \"ed5df646-0604-408b-bd6e-a502f0bad8e1\") " pod="openshift-network-operator/iptables-alerter-ksj4j"
Jul 16 00:40:29.850430 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.850393    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/090eddb9-3f4c-4a94-9b62-4f6bf8f38428-host-var-lib-cni-bin\") pod \"multus-h4ks8\" (UID: \"090eddb9-3f4c-4a94-9b62-4f6bf8f38428\") " pod="openshift-multus/multus-h4ks8"
Jul 16 00:40:29.850430 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.850417    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/090eddb9-3f4c-4a94-9b62-4f6bf8f38428-multus-conf-dir\") pod \"multus-h4ks8\" (UID: \"090eddb9-3f4c-4a94-9b62-4f6bf8f38428\") " pod="openshift-multus/multus-h4ks8"
Jul 16 00:40:29.850832 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.850440    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/090eddb9-3f4c-4a94-9b62-4f6bf8f38428-host-run-multus-certs\") pod \"multus-h4ks8\" (UID: \"090eddb9-3f4c-4a94-9b62-4f6bf8f38428\") " pod="openshift-multus/multus-h4ks8"
Jul 16 00:40:29.850832 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.850463    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/090eddb9-3f4c-4a94-9b62-4f6bf8f38428-etc-kubernetes\") pod \"multus-h4ks8\" (UID: \"090eddb9-3f4c-4a94-9b62-4f6bf8f38428\") " pod="openshift-multus/multus-h4ks8"
Jul 16 00:40:29.850832 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.850489    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ed5df646-0604-408b-bd6e-a502f0bad8e1-iptables-alerter-script\") pod \"iptables-alerter-ksj4j\" (UID: \"ed5df646-0604-408b-bd6e-a502f0bad8e1\") " pod="openshift-network-operator/iptables-alerter-ksj4j"
Jul 16 00:40:29.850832 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.850516    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/090eddb9-3f4c-4a94-9b62-4f6bf8f38428-host-run-k8s-cni-cncf-io\") pod \"multus-h4ks8\" (UID: \"090eddb9-3f4c-4a94-9b62-4f6bf8f38428\") " pod="openshift-multus/multus-h4ks8"
Jul 16 00:40:29.850832 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.850541    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/090eddb9-3f4c-4a94-9b62-4f6bf8f38428-multus-daemon-config\") pod \"multus-h4ks8\" (UID: \"090eddb9-3f4c-4a94-9b62-4f6bf8f38428\") " pod="openshift-multus/multus-h4ks8"
Jul 16 00:40:29.850832 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.850600    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ed5df646-0604-408b-bd6e-a502f0bad8e1-host-slash\") pod \"iptables-alerter-ksj4j\" (UID: \"ed5df646-0604-408b-bd6e-a502f0bad8e1\") " pod="openshift-network-operator/iptables-alerter-ksj4j"
Jul 16 00:40:29.850832 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.850674    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/090eddb9-3f4c-4a94-9b62-4f6bf8f38428-system-cni-dir\") pod \"multus-h4ks8\" (UID: \"090eddb9-3f4c-4a94-9b62-4f6bf8f38428\") " pod="openshift-multus/multus-h4ks8"
Jul 16 00:40:29.850832 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.850710    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/090eddb9-3f4c-4a94-9b62-4f6bf8f38428-cni-binary-copy\") pod \"multus-h4ks8\" (UID: \"090eddb9-3f4c-4a94-9b62-4f6bf8f38428\") " pod="openshift-multus/multus-h4ks8"
Jul 16 00:40:29.851583 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.851559    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 16 00:40:29.851851 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.851834    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hgcmx"
Jul 16 00:40:29.851929 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.851853    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 16 00:40:29.852017 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:40:29.851936    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hgcmx" podUID="16a7fa19-0ce5-49ca-96ef-dde5ae694dc0"
Jul 16 00:40:29.852340 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.852317    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 16 00:40:29.854729 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.854709    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-bn5hr"
Jul 16 00:40:29.857285 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.857225    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-l97c7"
Jul 16 00:40:29.860196 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.860071    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-tm5f5"
Jul 16 00:40:29.865105 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.863641    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bkf2h"
Jul 16 00:40:29.865105 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:40:29.863741    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bkf2h" podUID="051e94d1-76fa-484b-b118-b29951f4c5ae"
Jul 16 00:40:29.865274 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.865185    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-btw92"
Jul 16 00:40:29.865332 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:40:29.865257    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-btw92" podUID="1a9f4afb-53aa-401c-9a57-0cfc7e2f6af0"
Jul 16 00:40:29.900198 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.900170    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 16 00:40:29.900328 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.900222    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 16 00:40:29.900328 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.900295    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 16 00:40:29.900474 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.900457    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 16 00:40:29.900688 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.900641    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 16 00:40:29.900783 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.900723    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 16 00:40:29.900783 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.900764    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 16 00:40:29.900902 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.900829    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-smd2h\""
Jul 16 00:40:29.901255 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.901107    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-6fk4l\""
Jul 16 00:40:29.901255 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.901204    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-jdplp\""
Jul 16 00:40:29.902497 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.902131    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 16 00:40:29.902497 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.902165    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 16 00:40:29.902497 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.902298    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 16 00:40:29.902497 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.902363    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-7w298\""
Jul 16 00:40:29.902497 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.902402    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 16 00:40:29.902859 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.902541    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 16 00:40:29.902859 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.902642    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 16 00:40:29.902859 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.902822    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 16 00:40:29.903242 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.903220    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-pvz4l\""
Jul 16 00:40:29.909553 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.909522    2563 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 16 00:40:29.933829 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.933804    2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-mxkk4"
Jul 16 00:40:29.939579 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.939554    2563 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 16 00:40:29.950415 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.950387    2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-mxkk4"
Jul 16 00:40:29.951550 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.951520    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/090eddb9-3f4c-4a94-9b62-4f6bf8f38428-system-cni-dir\") pod \"multus-h4ks8\" (UID: \"090eddb9-3f4c-4a94-9b62-4f6bf8f38428\") " pod="openshift-multus/multus-h4ks8"
Jul 16 00:40:29.951677 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.951558    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/090eddb9-3f4c-4a94-9b62-4f6bf8f38428-multus-socket-dir-parent\") pod \"multus-h4ks8\" (UID: \"090eddb9-3f4c-4a94-9b62-4f6bf8f38428\") " pod="openshift-multus/multus-h4ks8"
Jul 16 00:40:29.951677 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.951588    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5p2f\" (UniqueName: \"kubernetes.io/projected/e990a21d-614b-4d1d-be54-0df6e44487ed-kube-api-access-x5p2f\") pod \"ovnkube-node-bn5hr\" (UID: \"e990a21d-614b-4d1d-be54-0df6e44487ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn5hr"
Jul 16 00:40:29.951677 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.951607    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcsrk\" (UniqueName: \"kubernetes.io/projected/49429502-4592-4b95-a41b-7f831eff48da-kube-api-access-jcsrk\") pod \"multus-additional-cni-plugins-l97c7\" (UID: \"49429502-4592-4b95-a41b-7f831eff48da\") " pod="openshift-multus/multus-additional-cni-plugins-l97c7"
Jul 16 00:40:29.951677 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.951627    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5a58757b-87fa-4446-b066-015fb48797cf-kubelet-dir\") pod \"aws-ebs-csi-driver-node-sfpw7\" (UID: \"5a58757b-87fa-4446-b066-015fb48797cf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sfpw7"
Jul 16 00:40:29.951677 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.951650    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5a58757b-87fa-4446-b066-015fb48797cf-device-dir\") pod \"aws-ebs-csi-driver-node-sfpw7\" (UID: \"5a58757b-87fa-4446-b066-015fb48797cf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sfpw7"
Jul 16 00:40:29.951677 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.951662    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/090eddb9-3f4c-4a94-9b62-4f6bf8f38428-system-cni-dir\") pod \"multus-h4ks8\" (UID: \"090eddb9-3f4c-4a94-9b62-4f6bf8f38428\") " pod="openshift-multus/multus-h4ks8"
Jul 16 00:40:29.952032 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.951667    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5a58757b-87fa-4446-b066-015fb48797cf-etc-selinux\") pod \"aws-ebs-csi-driver-node-sfpw7\" (UID: \"5a58757b-87fa-4446-b066-015fb48797cf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sfpw7"
Jul 16 00:40:29.952032 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.951727    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/090eddb9-3f4c-4a94-9b62-4f6bf8f38428-multus-socket-dir-parent\") pod \"multus-h4ks8\" (UID: \"090eddb9-3f4c-4a94-9b62-4f6bf8f38428\") " pod="openshift-multus/multus-h4ks8"
Jul 16 00:40:29.952032 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.951765    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/090eddb9-3f4c-4a94-9b62-4f6bf8f38428-cnibin\") pod \"multus-h4ks8\" (UID: \"090eddb9-3f4c-4a94-9b62-4f6bf8f38428\") " pod="openshift-multus/multus-h4ks8"
Jul 16 00:40:29.952032 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.951792    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/24b5fdac-6964-47c8-aa9e-d268d052c2e3-lib-modules\") pod \"tuned-rjk6r\" (UID: \"24b5fdac-6964-47c8-aa9e-d268d052c2e3\") " pod="openshift-cluster-node-tuning-operator/tuned-rjk6r"
Jul 16 00:40:29.952032 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.951816    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/24b5fdac-6964-47c8-aa9e-d268d052c2e3-host\") pod \"tuned-rjk6r\" (UID: \"24b5fdac-6964-47c8-aa9e-d268d052c2e3\") " pod="openshift-cluster-node-tuning-operator/tuned-rjk6r"
Jul 16 00:40:29.952032 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.951837    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/090eddb9-3f4c-4a94-9b62-4f6bf8f38428-cnibin\") pod \"multus-h4ks8\" (UID: \"090eddb9-3f4c-4a94-9b62-4f6bf8f38428\") " pod="openshift-multus/multus-h4ks8"
Jul 16 00:40:29.952032 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.951841    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd81358e-a649-4ba2-b63e-d9ba85ee93db-tmp-dir\") pod \"node-resolver-gzbcl\" (UID: \"bd81358e-a649-4ba2-b63e-d9ba85ee93db\") " pod="openshift-dns/node-resolver-gzbcl"
Jul 16 00:40:29.952032 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.951880    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/49429502-4592-4b95-a41b-7f831eff48da-cni-binary-copy\") pod \"multus-additional-cni-plugins-l97c7\" (UID: \"49429502-4592-4b95-a41b-7f831eff48da\") " pod="openshift-multus/multus-additional-cni-plugins-l97c7"
Jul 16 00:40:29.952032 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.951925    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/090eddb9-3f4c-4a94-9b62-4f6bf8f38428-host-var-lib-cni-bin\") pod \"multus-h4ks8\" (UID: \"090eddb9-3f4c-4a94-9b62-4f6bf8f38428\") " pod="openshift-multus/multus-h4ks8"
Jul 16 00:40:29.952032 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.951950    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/090eddb9-3f4c-4a94-9b62-4f6bf8f38428-multus-conf-dir\") pod \"multus-h4ks8\" (UID: \"090eddb9-3f4c-4a94-9b62-4f6bf8f38428\") " pod="openshift-multus/multus-h4ks8"
Jul 16 00:40:29.952032 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.951963    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/090eddb9-3f4c-4a94-9b62-4f6bf8f38428-host-var-lib-cni-bin\") pod \"multus-h4ks8\" (UID: \"090eddb9-3f4c-4a94-9b62-4f6bf8f38428\") " pod="openshift-multus/multus-h4ks8"
Jul 16 00:40:29.952032 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.951976    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/090eddb9-3f4c-4a94-9b62-4f6bf8f38428-etc-kubernetes\") pod \"multus-h4ks8\" (UID: \"090eddb9-3f4c-4a94-9b62-4f6bf8f38428\") " pod="openshift-multus/multus-h4ks8"
Jul 16 00:40:29.952032 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.952001    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e990a21d-614b-4d1d-be54-0df6e44487ed-env-overrides\") pod \"ovnkube-node-bn5hr\" (UID: \"e990a21d-614b-4d1d-be54-0df6e44487ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn5hr"
Jul 16 00:40:29.952032 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.952023    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f0b6d16f-9a13-480c-b981-6481983dbd91-host\") pod \"node-ca-rk85k\" (UID: \"f0b6d16f-9a13-480c-b981-6481983dbd91\") " pod="openshift-image-registry/node-ca-rk85k"
Jul 16 00:40:29.952032 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.952024    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/090eddb9-3f4c-4a94-9b62-4f6bf8f38428-multus-conf-dir\") pod \"multus-h4ks8\" (UID: \"090eddb9-3f4c-4a94-9b62-4f6bf8f38428\") " pod="openshift-multus/multus-h4ks8"
Jul 16 00:40:29.952830 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.952063    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/090eddb9-3f4c-4a94-9b62-4f6bf8f38428-etc-kubernetes\") pod \"multus-h4ks8\" (UID: \"090eddb9-3f4c-4a94-9b62-4f6bf8f38428\") " pod="openshift-multus/multus-h4ks8"
Jul 16 00:40:29.952830 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.952096    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e990a21d-614b-4d1d-be54-0df6e44487ed-systemd-units\") pod \"ovnkube-node-bn5hr\" (UID: \"e990a21d-614b-4d1d-be54-0df6e44487ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn5hr"
Jul 16 00:40:29.952830 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.952128    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e990a21d-614b-4d1d-be54-0df6e44487ed-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bn5hr\" (UID: \"e990a21d-614b-4d1d-be54-0df6e44487ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn5hr"
Jul 16 00:40:29.952830 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.952157    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqw59\" (UniqueName: \"kubernetes.io/projected/f0b6d16f-9a13-480c-b981-6481983dbd91-kube-api-access-xqw59\") pod \"node-ca-rk85k\" (UID: \"f0b6d16f-9a13-480c-b981-6481983dbd91\") " pod="openshift-image-registry/node-ca-rk85k"
Jul 16 00:40:29.952830 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.952189    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/090eddb9-3f4c-4a94-9b62-4f6bf8f38428-host-var-lib-cni-multus\") pod \"multus-h4ks8\" (UID: \"090eddb9-3f4c-4a94-9b62-4f6bf8f38428\") " pod="openshift-multus/multus-h4ks8"
Jul 16 00:40:29.952830 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.952215    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e990a21d-614b-4d1d-be54-0df6e44487ed-etc-openvswitch\") pod \"ovnkube-node-bn5hr\" (UID: \"e990a21d-614b-4d1d-be54-0df6e44487ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn5hr"
Jul 16 00:40:29.952830 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.952240    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/49429502-4592-4b95-a41b-7f831eff48da-os-release\") pod \"multus-additional-cni-plugins-l97c7\" (UID: \"49429502-4592-4b95-a41b-7f831eff48da\") " pod="openshift-multus/multus-additional-cni-plugins-l97c7"
Jul 16 00:40:29.952830 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.952276    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/39f3a671-1715-4dd3-9ca0-0b714dde3281-konnectivity-ca\") pod \"konnectivity-agent-tm5f5\" (UID: \"39f3a671-1715-4dd3-9ca0-0b714dde3281\") " pod="kube-system/konnectivity-agent-tm5f5"
Jul 16 00:40:29.952830 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.952278    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/090eddb9-3f4c-4a94-9b62-4f6bf8f38428-host-var-lib-cni-multus\") pod \"multus-h4ks8\" (UID: \"090eddb9-3f4c-4a94-9b62-4f6bf8f38428\") " pod="openshift-multus/multus-h4ks8"
Jul 16 00:40:29.952830 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.952301    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/24b5fdac-6964-47c8-aa9e-d268d052c2e3-etc-tuned\") pod \"tuned-rjk6r\" (UID: \"24b5fdac-6964-47c8-aa9e-d268d052c2e3\") " pod="openshift-cluster-node-tuning-operator/tuned-rjk6r"
Jul 16 00:40:29.952830 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.952349    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/090eddb9-3f4c-4a94-9b62-4f6bf8f38428-multus-cni-dir\") pod \"multus-h4ks8\" (UID: \"090eddb9-3f4c-4a94-9b62-4f6bf8f38428\") " pod="openshift-multus/multus-h4ks8"
Jul 16 00:40:29.952830 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.952410    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mz4pp\" (UniqueName: \"kubernetes.io/projected/090eddb9-3f4c-4a94-9b62-4f6bf8f38428-kube-api-access-mz4pp\") pod \"multus-h4ks8\" (UID: \"090eddb9-3f4c-4a94-9b62-4f6bf8f38428\") " pod="openshift-multus/multus-h4ks8"
Jul 16 00:40:29.952830 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.952421    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/090eddb9-3f4c-4a94-9b62-4f6bf8f38428-multus-cni-dir\") pod \"multus-h4ks8\" (UID: \"090eddb9-3f4c-4a94-9b62-4f6bf8f38428\") " pod="openshift-multus/multus-h4ks8"
Jul 16 00:40:29.952830 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.952456    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/16a7fa19-0ce5-49ca-96ef-dde5ae694dc0-original-pull-secret\") pod \"global-pull-secret-syncer-hgcmx\" (UID: \"16a7fa19-0ce5-49ca-96ef-dde5ae694dc0\") " pod="kube-system/global-pull-secret-syncer-hgcmx"
Jul 16 00:40:29.952830 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.952481    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e990a21d-614b-4d1d-be54-0df6e44487ed-host-run-netns\") pod \"ovnkube-node-bn5hr\" (UID: \"e990a21d-614b-4d1d-be54-0df6e44487ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn5hr"
Jul 16 00:40:29.952830 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.952504    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e990a21d-614b-4d1d-be54-0df6e44487ed-host-run-ovn-kubernetes\") pod \"ovnkube-node-bn5hr\" (UID: \"e990a21d-614b-4d1d-be54-0df6e44487ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn5hr"
Jul 16 00:40:29.952830 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.952535    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7j7f\" (UniqueName: \"kubernetes.io/projected/1a9f4afb-53aa-401c-9a57-0cfc7e2f6af0-kube-api-access-t7j7f\") pod \"network-check-target-btw92\" (UID: \"1a9f4afb-53aa-401c-9a57-0cfc7e2f6af0\") " pod="openshift-network-diagnostics/network-check-target-btw92"
Jul 16 00:40:29.953790 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.952585    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7npff\" (UniqueName: \"kubernetes.io/projected/5a58757b-87fa-4446-b066-015fb48797cf-kube-api-access-7npff\") pod \"aws-ebs-csi-driver-node-sfpw7\" (UID: \"5a58757b-87fa-4446-b066-015fb48797cf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sfpw7"
Jul 16 00:40:29.953790 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.952629    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/24b5fdac-6964-47c8-aa9e-d268d052c2e3-etc-sysctl-d\") pod \"tuned-rjk6r\" (UID: \"24b5fdac-6964-47c8-aa9e-d268d052c2e3\") " pod="openshift-cluster-node-tuning-operator/tuned-rjk6r"
Jul 16 00:40:29.953790 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.952661    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e990a21d-614b-4d1d-be54-0df6e44487ed-log-socket\") pod \"ovnkube-node-bn5hr\" (UID: \"e990a21d-614b-4d1d-be54-0df6e44487ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn5hr"
Jul 16 00:40:29.953790 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.952689    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/49429502-4592-4b95-a41b-7f831eff48da-system-cni-dir\") pod \"multus-additional-cni-plugins-l97c7\" (UID: \"49429502-4592-4b95-a41b-7f831eff48da\") " pod="openshift-multus/multus-additional-cni-plugins-l97c7"
Jul 16 00:40:29.953790 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.952716    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/24b5fdac-6964-47c8-aa9e-d268d052c2e3-etc-sysconfig\") pod \"tuned-rjk6r\" (UID: \"24b5fdac-6964-47c8-aa9e-d268d052c2e3\") " pod="openshift-cluster-node-tuning-operator/tuned-rjk6r"
Jul 16 00:40:29.953790 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.952739    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/24b5fdac-6964-47c8-aa9e-d268d052c2e3-run\") pod \"tuned-rjk6r\" (UID: \"24b5fdac-6964-47c8-aa9e-d268d052c2e3\") " pod="openshift-cluster-node-tuning-operator/tuned-rjk6r"
Jul 16 00:40:29.953790 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.952804    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/24b5fdac-6964-47c8-aa9e-d268d052c2e3-sys\") pod \"tuned-rjk6r\" (UID: \"24b5fdac-6964-47c8-aa9e-d268d052c2e3\") " pod="openshift-cluster-node-tuning-operator/tuned-rjk6r"
Jul 16 00:40:29.953790 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.952836    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/090eddb9-3f4c-4a94-9b62-4f6bf8f38428-host-run-k8s-cni-cncf-io\") pod \"multus-h4ks8\" (UID: \"090eddb9-3f4c-4a94-9b62-4f6bf8f38428\") " pod="openshift-multus/multus-h4ks8"
Jul 16 00:40:29.953790 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.952866    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/16a7fa19-0ce5-49ca-96ef-dde5ae694dc0-kubelet-config\") pod \"global-pull-secret-syncer-hgcmx\" (UID: \"16a7fa19-0ce5-49ca-96ef-dde5ae694dc0\") " pod="kube-system/global-pull-secret-syncer-hgcmx"
Jul 16 00:40:29.953790 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.952901    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/090eddb9-3f4c-4a94-9b62-4f6bf8f38428-host-run-k8s-cni-cncf-io\") pod \"multus-h4ks8\" (UID: \"090eddb9-3f4c-4a94-9b62-4f6bf8f38428\") " pod="openshift-multus/multus-h4ks8"
Jul 16 00:40:29.953790 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.952904    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvkbs\" (UniqueName: \"kubernetes.io/projected/bd81358e-a649-4ba2-b63e-d9ba85ee93db-kube-api-access-lvkbs\") pod \"node-resolver-gzbcl\" (UID: \"bd81358e-a649-4ba2-b63e-d9ba85ee93db\") " pod="openshift-dns/node-resolver-gzbcl"
Jul 16 00:40:29.953790 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.952960    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/090eddb9-3f4c-4a94-9b62-4f6bf8f38428-host-var-lib-kubelet\") pod \"multus-h4ks8\" (UID: \"090eddb9-3f4c-4a94-9b62-4f6bf8f38428\") " pod="openshift-multus/multus-h4ks8"
Jul 16 00:40:29.953790 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.952989    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e990a21d-614b-4d1d-be54-0df6e44487ed-ovnkube-script-lib\") pod \"ovnkube-node-bn5hr\" (UID: \"e990a21d-614b-4d1d-be54-0df6e44487ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn5hr"
Jul 16 00:40:29.953790 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.953014    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/24b5fdac-6964-47c8-aa9e-d268d052c2e3-etc-sysctl-conf\") pod \"tuned-rjk6r\" (UID: \"24b5fdac-6964-47c8-aa9e-d268d052c2e3\") " pod="openshift-cluster-node-tuning-operator/tuned-rjk6r"
Jul 16 00:40:29.953790 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.953052    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e990a21d-614b-4d1d-be54-0df6e44487ed-ovnkube-config\") pod \"ovnkube-node-bn5hr\" (UID: \"e990a21d-614b-4d1d-be54-0df6e44487ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn5hr"
Jul 16 00:40:29.953790 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.953063    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/090eddb9-3f4c-4a94-9b62-4f6bf8f38428-host-var-lib-kubelet\") pod \"multus-h4ks8\" (UID: \"090eddb9-3f4c-4a94-9b62-4f6bf8f38428\") " pod="openshift-multus/multus-h4ks8"
Jul 16 00:40:29.953790 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.953102    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/24b5fdac-6964-47c8-aa9e-d268d052c2e3-etc-systemd\") pod \"tuned-rjk6r\" (UID: \"24b5fdac-6964-47c8-aa9e-d268d052c2e3\") " pod="openshift-cluster-node-tuning-operator/tuned-rjk6r"
Jul 16 00:40:29.954763 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.953128    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/24b5fdac-6964-47c8-aa9e-d268d052c2e3-tmp\") pod \"tuned-rjk6r\" (UID: \"24b5fdac-6964-47c8-aa9e-d268d052c2e3\") " pod="openshift-cluster-node-tuning-operator/tuned-rjk6r"
Jul 16 00:40:29.954763 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.953155    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/49429502-4592-4b95-a41b-7f831eff48da-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-l97c7\" (UID: \"49429502-4592-4b95-a41b-7f831eff48da\") " pod="openshift-multus/multus-additional-cni-plugins-l97c7"
Jul 16 00:40:29.954763 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.953194    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/090eddb9-3f4c-4a94-9b62-4f6bf8f38428-host-run-multus-certs\") pod \"multus-h4ks8\" (UID: \"090eddb9-3f4c-4a94-9b62-4f6bf8f38428\") " pod="openshift-multus/multus-h4ks8"
Jul 16 00:40:29.954763 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.953221    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/16a7fa19-0ce5-49ca-96ef-dde5ae694dc0-dbus\") pod \"global-pull-secret-syncer-hgcmx\" (UID: \"16a7fa19-0ce5-49ca-96ef-dde5ae694dc0\") " pod="kube-system/global-pull-secret-syncer-hgcmx"
Jul 16 00:40:29.954763 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.953265    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/090eddb9-3f4c-4a94-9b62-4f6bf8f38428-host-run-multus-certs\") pod \"multus-h4ks8\" (UID: \"090eddb9-3f4c-4a94-9b62-4f6bf8f38428\") " pod="openshift-multus/multus-h4ks8"
Jul 16 00:40:29.954763 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.953296    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e990a21d-614b-4d1d-be54-0df6e44487ed-host-kubelet\") pod \"ovnkube-node-bn5hr\" (UID: \"e990a21d-614b-4d1d-be54-0df6e44487ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn5hr"
Jul 16 00:40:29.954763 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.953321    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e990a21d-614b-4d1d-be54-0df6e44487ed-var-lib-openvswitch\") pod \"ovnkube-node-bn5hr\" (UID: \"e990a21d-614b-4d1d-be54-0df6e44487ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn5hr"
Jul 16 00:40:29.954763 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.953364    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e990a21d-614b-4d1d-be54-0df6e44487ed-run-ovn\") pod \"ovnkube-node-bn5hr\" (UID: \"e990a21d-614b-4d1d-be54-0df6e44487ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn5hr"
Jul 16 00:40:29.954763 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.953397    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e990a21d-614b-4d1d-be54-0df6e44487ed-host-cni-bin\") pod \"ovnkube-node-bn5hr\" (UID: \"e990a21d-614b-4d1d-be54-0df6e44487ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn5hr"
Jul 16 00:40:29.954763 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.953426    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/49429502-4592-4b95-a41b-7f831eff48da-cnibin\") pod \"multus-additional-cni-plugins-l97c7\" (UID: \"49429502-4592-4b95-a41b-7f831eff48da\") " pod="openshift-multus/multus-additional-cni-plugins-l97c7"
Jul 16 00:40:29.954763 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.953452    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5a58757b-87fa-4446-b066-015fb48797cf-sys-fs\") pod \"aws-ebs-csi-driver-node-sfpw7\" (UID: \"5a58757b-87fa-4446-b066-015fb48797cf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sfpw7"
Jul 16 00:40:29.954763 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.953475    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/24b5fdac-6964-47c8-aa9e-d268d052c2e3-etc-modprobe-d\") pod \"tuned-rjk6r\" (UID: \"24b5fdac-6964-47c8-aa9e-d268d052c2e3\") " pod="openshift-cluster-node-tuning-operator/tuned-rjk6r"
Jul 16 00:40:29.954763 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.953499    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrq8r\" (UniqueName: \"kubernetes.io/projected/24b5fdac-6964-47c8-aa9e-d268d052c2e3-kube-api-access-jrq8r\") pod \"tuned-rjk6r\" (UID: \"24b5fdac-6964-47c8-aa9e-d268d052c2e3\") " pod="openshift-cluster-node-tuning-operator/tuned-rjk6r"
Jul 16 00:40:29.954763 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.953528    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/090eddb9-3f4c-4a94-9b62-4f6bf8f38428-cni-binary-copy\") pod \"multus-h4ks8\" (UID: \"090eddb9-3f4c-4a94-9b62-4f6bf8f38428\") " pod="openshift-multus/multus-h4ks8"
Jul 16 00:40:29.954763 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.953556    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/090eddb9-3f4c-4a94-9b62-4f6bf8f38428-host-run-netns\") pod \"multus-h4ks8\" (UID: \"090eddb9-3f4c-4a94-9b62-4f6bf8f38428\") " pod="openshift-multus/multus-h4ks8"
Jul 16 00:40:29.954763 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.953580    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e990a21d-614b-4d1d-be54-0df6e44487ed-ovn-node-metrics-cert\") pod \"ovnkube-node-bn5hr\" (UID: \"e990a21d-614b-4d1d-be54-0df6e44487ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn5hr"
Jul 16 00:40:29.955645 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.953606    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/49429502-4592-4b95-a41b-7f831eff48da-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-l97c7\" (UID: \"49429502-4592-4b95-a41b-7f831eff48da\") " pod="openshift-multus/multus-additional-cni-plugins-l97c7"
Jul 16 00:40:29.955645 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.953632    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/24b5fdac-6964-47c8-aa9e-d268d052c2e3-etc-kubernetes\") pod \"tuned-rjk6r\" (UID: \"24b5fdac-6964-47c8-aa9e-d268d052c2e3\") " pod="openshift-cluster-node-tuning-operator/tuned-rjk6r"
Jul 16 00:40:29.955645 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.953652    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/bd81358e-a649-4ba2-b63e-d9ba85ee93db-hosts-file\") pod \"node-resolver-gzbcl\" (UID: \"bd81358e-a649-4ba2-b63e-d9ba85ee93db\") " pod="openshift-dns/node-resolver-gzbcl"
Jul 16 00:40:29.955645 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.953668    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/090eddb9-3f4c-4a94-9b62-4f6bf8f38428-os-release\") pod \"multus-h4ks8\" (UID: \"090eddb9-3f4c-4a94-9b62-4f6bf8f38428\") " pod="openshift-multus/multus-h4ks8"
Jul 16 00:40:29.955645 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.953691    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/090eddb9-3f4c-4a94-9b62-4f6bf8f38428-hostroot\") pod \"multus-h4ks8\" (UID: \"090eddb9-3f4c-4a94-9b62-4f6bf8f38428\") " pod="openshift-multus/multus-h4ks8"
Jul 16 00:40:29.955645 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.953717    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlpjz\" (UniqueName: \"kubernetes.io/projected/ed5df646-0604-408b-bd6e-a502f0bad8e1-kube-api-access-qlpjz\") pod \"iptables-alerter-ksj4j\" (UID: \"ed5df646-0604-408b-bd6e-a502f0bad8e1\") " pod="openshift-network-operator/iptables-alerter-ksj4j"
Jul 16 00:40:29.955645 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.953744    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e990a21d-614b-4d1d-be54-0df6e44487ed-host-cni-netd\") pod \"ovnkube-node-bn5hr\" (UID: \"e990a21d-614b-4d1d-be54-0df6e44487ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn5hr"
Jul 16 00:40:29.955645 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.953768    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/39f3a671-1715-4dd3-9ca0-0b714dde3281-agent-certs\") pod \"konnectivity-agent-tm5f5\" (UID: \"39f3a671-1715-4dd3-9ca0-0b714dde3281\") " pod="kube-system/konnectivity-agent-tm5f5"
Jul 16 00:40:29.955645 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.953793    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5a58757b-87fa-4446-b066-015fb48797cf-registration-dir\") pod \"aws-ebs-csi-driver-node-sfpw7\" (UID: \"5a58757b-87fa-4446-b066-015fb48797cf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sfpw7"
Jul 16 00:40:29.955645 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.953818    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ed5df646-0604-408b-bd6e-a502f0bad8e1-iptables-alerter-script\") pod \"iptables-alerter-ksj4j\" (UID: \"ed5df646-0604-408b-bd6e-a502f0bad8e1\") " pod="openshift-network-operator/iptables-alerter-ksj4j"
Jul 16 00:40:29.955645 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.953867    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e990a21d-614b-4d1d-be54-0df6e44487ed-run-systemd\") pod \"ovnkube-node-bn5hr\" (UID: \"e990a21d-614b-4d1d-be54-0df6e44487ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn5hr"
Jul 16 00:40:29.955645 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.953892    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e990a21d-614b-4d1d-be54-0df6e44487ed-run-openvswitch\") pod \"ovnkube-node-bn5hr\" (UID: \"e990a21d-614b-4d1d-be54-0df6e44487ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn5hr"
Jul 16 00:40:29.955645 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.953919    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/49429502-4592-4b95-a41b-7f831eff48da-tuning-conf-dir\") pod \"multus-additional-cni-plugins-l97c7\" (UID: \"49429502-4592-4b95-a41b-7f831eff48da\") " pod="openshift-multus/multus-additional-cni-plugins-l97c7"
Jul 16 00:40:29.955645 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.953973    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/051e94d1-76fa-484b-b118-b29951f4c5ae-metrics-certs\") pod \"network-metrics-daemon-bkf2h\" (UID: \"051e94d1-76fa-484b-b118-b29951f4c5ae\") " pod="openshift-multus/network-metrics-daemon-bkf2h"
Jul 16 00:40:29.955645 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.954015    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvtjv\" (UniqueName: \"kubernetes.io/projected/051e94d1-76fa-484b-b118-b29951f4c5ae-kube-api-access-wvtjv\") pod \"network-metrics-daemon-bkf2h\" (UID: \"051e94d1-76fa-484b-b118-b29951f4c5ae\") " pod="openshift-multus/network-metrics-daemon-bkf2h"
Jul 16 00:40:29.955645 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.954042    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/24b5fdac-6964-47c8-aa9e-d268d052c2e3-var-lib-kubelet\") pod \"tuned-rjk6r\" (UID: \"24b5fdac-6964-47c8-aa9e-d268d052c2e3\") " pod="openshift-cluster-node-tuning-operator/tuned-rjk6r"
Jul 16 00:40:29.956607 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.954082    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f0b6d16f-9a13-480c-b981-6481983dbd91-serviceca\") pod \"node-ca-rk85k\" (UID: \"f0b6d16f-9a13-480c-b981-6481983dbd91\") " pod="openshift-image-registry/node-ca-rk85k"
Jul 16 00:40:29.956607 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.954109    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/090eddb9-3f4c-4a94-9b62-4f6bf8f38428-multus-daemon-config\") pod \"multus-h4ks8\" (UID: \"090eddb9-3f4c-4a94-9b62-4f6bf8f38428\") " pod="openshift-multus/multus-h4ks8"
Jul 16 00:40:29.956607 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.954127    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ed5df646-0604-408b-bd6e-a502f0bad8e1-host-slash\") pod \"iptables-alerter-ksj4j\" (UID: \"ed5df646-0604-408b-bd6e-a502f0bad8e1\") " pod="openshift-network-operator/iptables-alerter-ksj4j"
Jul 16 00:40:29.956607 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.954143    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e990a21d-614b-4d1d-be54-0df6e44487ed-host-slash\") pod \"ovnkube-node-bn5hr\" (UID: \"e990a21d-614b-4d1d-be54-0df6e44487ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn5hr"
Jul 16 00:40:29.956607 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.954160    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e990a21d-614b-4d1d-be54-0df6e44487ed-node-log\") pod \"ovnkube-node-bn5hr\" (UID: \"e990a21d-614b-4d1d-be54-0df6e44487ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn5hr"
Jul 16 00:40:29.956607 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.954166    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/090eddb9-3f4c-4a94-9b62-4f6bf8f38428-cni-binary-copy\") pod \"multus-h4ks8\" (UID: \"090eddb9-3f4c-4a94-9b62-4f6bf8f38428\") " pod="openshift-multus/multus-h4ks8"
Jul 16 00:40:29.956607 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.954175    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5a58757b-87fa-4446-b066-015fb48797cf-socket-dir\") pod \"aws-ebs-csi-driver-node-sfpw7\" (UID: \"5a58757b-87fa-4446-b066-015fb48797cf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sfpw7"
Jul 16 00:40:29.956607 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.954226    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/090eddb9-3f4c-4a94-9b62-4f6bf8f38428-host-run-netns\") pod \"multus-h4ks8\" (UID: \"090eddb9-3f4c-4a94-9b62-4f6bf8f38428\") " pod="openshift-multus/multus-h4ks8"
Jul 16 00:40:29.956607 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.954346    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/090eddb9-3f4c-4a94-9b62-4f6bf8f38428-os-release\") pod \"multus-h4ks8\" (UID: \"090eddb9-3f4c-4a94-9b62-4f6bf8f38428\") " pod="openshift-multus/multus-h4ks8"
Jul 16 00:40:29.956607 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.954386    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/090eddb9-3f4c-4a94-9b62-4f6bf8f38428-hostroot\") pod \"multus-h4ks8\" (UID: \"090eddb9-3f4c-4a94-9b62-4f6bf8f38428\") " pod="openshift-multus/multus-h4ks8"
Jul 16 00:40:29.956607 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.954660    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ed5df646-0604-408b-bd6e-a502f0bad8e1-host-slash\") pod \"iptables-alerter-ksj4j\" (UID: \"ed5df646-0604-408b-bd6e-a502f0bad8e1\") " pod="openshift-network-operator/iptables-alerter-ksj4j"
Jul 16 00:40:29.956607 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.954857    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ed5df646-0604-408b-bd6e-a502f0bad8e1-iptables-alerter-script\") pod \"iptables-alerter-ksj4j\" (UID: \"ed5df646-0604-408b-bd6e-a502f0bad8e1\") " pod="openshift-network-operator/iptables-alerter-ksj4j"
Jul 16 00:40:29.956607 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.955124    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/090eddb9-3f4c-4a94-9b62-4f6bf8f38428-multus-daemon-config\") pod \"multus-h4ks8\" (UID: \"090eddb9-3f4c-4a94-9b62-4f6bf8f38428\") " pod="openshift-multus/multus-h4ks8"
Jul 16 00:40:29.960707 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.960675    2563 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 16 00:40:29.965380 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.965349    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mz4pp\" (UniqueName: \"kubernetes.io/projected/090eddb9-3f4c-4a94-9b62-4f6bf8f38428-kube-api-access-mz4pp\") pod \"multus-h4ks8\" (UID: \"090eddb9-3f4c-4a94-9b62-4f6bf8f38428\") " pod="openshift-multus/multus-h4ks8"
Jul 16 00:40:29.965604 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:29.965581    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlpjz\" (UniqueName: \"kubernetes.io/projected/ed5df646-0604-408b-bd6e-a502f0bad8e1-kube-api-access-qlpjz\") pod \"iptables-alerter-ksj4j\" (UID: \"ed5df646-0604-408b-bd6e-a502f0bad8e1\") " pod="openshift-network-operator/iptables-alerter-ksj4j"
Jul 16 00:40:30.005059 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:30.004964    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-65.ec2.internal" event={"ID":"69184a0cad720aed3f473960e4918c4c","Type":"ContainerStarted","Data":"a43ad39d145e33df585a14c4239d0c7bff30e91135d8e35675f1ac5e205d7a2d"}
Jul 16 00:40:30.006162 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:30.006137    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-65.ec2.internal" event={"ID":"ebeafacc0368100d06658fd0ec2ae27a","Type":"ContainerStarted","Data":"4f179a00f926e3186e2b2244e4433625c3c802f3048d5e7cd05caec34ccf4fbd"}
Jul 16 00:40:30.055193 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:30.055159    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e990a21d-614b-4d1d-be54-0df6e44487ed-etc-openvswitch\") pod \"ovnkube-node-bn5hr\" (UID: \"e990a21d-614b-4d1d-be54-0df6e44487ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn5hr"
Jul 16 00:40:30.055193 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:30.055200    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/49429502-4592-4b95-a41b-7f831eff48da-os-release\") pod \"multus-additional-cni-plugins-l97c7\" (UID: \"49429502-4592-4b95-a41b-7f831eff48da\") " pod="openshift-multus/multus-additional-cni-plugins-l97c7"
Jul 16 00:40:30.055381 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:30.055218    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/39f3a671-1715-4dd3-9ca0-0b714dde3281-konnectivity-ca\") pod \"konnectivity-agent-tm5f5\" (UID: \"39f3a671-1715-4dd3-9ca0-0b714dde3281\") " pod="kube-system/konnectivity-agent-tm5f5"
Jul 16 00:40:30.055381 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:30.055243    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/24b5fdac-6964-47c8-aa9e-d268d052c2e3-etc-tuned\") pod \"tuned-rjk6r\" (UID: \"24b5fdac-6964-47c8-aa9e-d268d052c2e3\") " pod="openshift-cluster-node-tuning-operator/tuned-rjk6r"
Jul 16 00:40:30.055381 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:30.055281    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e990a21d-614b-4d1d-be54-0df6e44487ed-etc-openvswitch\") pod \"ovnkube-node-bn5hr\" (UID: \"e990a21d-614b-4d1d-be54-0df6e44487ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn5hr"
Jul 16 00:40:30.055381 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:30.055299    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/16a7fa19-0ce5-49ca-96ef-dde5ae694dc0-original-pull-secret\") pod \"global-pull-secret-syncer-hgcmx\" (UID: \"16a7fa19-0ce5-49ca-96ef-dde5ae694dc0\") " pod="kube-system/global-pull-secret-syncer-hgcmx"
Jul 16 00:40:30.055381 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:30.055338    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e990a21d-614b-4d1d-be54-0df6e44487ed-host-run-netns\") pod \"ovnkube-node-bn5hr\" (UID: \"e990a21d-614b-4d1d-be54-0df6e44487ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn5hr"
Jul 16 00:40:30.055381 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:30.055360    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/49429502-4592-4b95-a41b-7f831eff48da-os-release\") pod \"multus-additional-cni-plugins-l97c7\" (UID: \"49429502-4592-4b95-a41b-7f831eff48da\") " pod="openshift-multus/multus-additional-cni-plugins-l97c7"
Jul 16 00:40:30.055381 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:30.055365    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e990a21d-614b-4d1d-be54-0df6e44487ed-host-run-ovn-kubernetes\") pod \"ovnkube-node-bn5hr\" (UID: \"e990a21d-614b-4d1d-be54-0df6e44487ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn5hr"
Jul 16 00:40:30.055793 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:30.055397    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7j7f\" (UniqueName: \"kubernetes.io/projected/1a9f4afb-53aa-401c-9a57-0cfc7e2f6af0-kube-api-access-t7j7f\") pod \"network-check-target-btw92\" (UID: \"1a9f4afb-53aa-401c-9a57-0cfc7e2f6af0\") " pod="openshift-network-diagnostics/network-check-target-btw92"
Jul 16 00:40:30.055793 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:30.055412    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e990a21d-614b-4d1d-be54-0df6e44487ed-host-run-netns\") pod \"ovnkube-node-bn5hr\" (UID: \"e990a21d-614b-4d1d-be54-0df6e44487ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn5hr"
Jul 16 00:40:30.055793 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:30.055424    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7npff\" (UniqueName: \"kubernetes.io/projected/5a58757b-87fa-4446-b066-015fb48797cf-kube-api-access-7npff\") pod \"aws-ebs-csi-driver-node-sfpw7\" (UID: \"5a58757b-87fa-4446-b066-015fb48797cf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sfpw7"
Jul 16 00:40:30.055793 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:40:30.055447    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 16 00:40:30.055793 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:40:30.055532    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16a7fa19-0ce5-49ca-96ef-dde5ae694dc0-original-pull-secret podName:16a7fa19-0ce5-49ca-96ef-dde5ae694dc0 nodeName:}" failed. No retries permitted until 2026-07-16 00:40:30.555489747 +0000 UTC m=+2.990751486 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/16a7fa19-0ce5-49ca-96ef-dde5ae694dc0-original-pull-secret") pod "global-pull-secret-syncer-hgcmx" (UID: "16a7fa19-0ce5-49ca-96ef-dde5ae694dc0") : object "kube-system"/"original-pull-secret" not registered
Jul 16 00:40:30.055793 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:30.055547    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e990a21d-614b-4d1d-be54-0df6e44487ed-host-run-ovn-kubernetes\") pod \"ovnkube-node-bn5hr\" (UID: \"e990a21d-614b-4d1d-be54-0df6e44487ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn5hr"
Jul 16 00:40:30.055793 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:30.055574    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/24b5fdac-6964-47c8-aa9e-d268d052c2e3-etc-sysctl-d\") pod \"tuned-rjk6r\" (UID: \"24b5fdac-6964-47c8-aa9e-d268d052c2e3\") " pod="openshift-cluster-node-tuning-operator/tuned-rjk6r"
Jul 16 00:40:30.055793 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:30.055448    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/24b5fdac-6964-47c8-aa9e-d268d052c2e3-etc-sysctl-d\") pod \"tuned-rjk6r\" (UID: \"24b5fdac-6964-47c8-aa9e-d268d052c2e3\") " pod="openshift-cluster-node-tuning-operator/tuned-rjk6r"
Jul 16 00:40:30.055793 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:30.055613    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e990a21d-614b-4d1d-be54-0df6e44487ed-log-socket\") pod \"ovnkube-node-bn5hr\" (UID: \"e990a21d-614b-4d1d-be54-0df6e44487ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn5hr"
Jul 16 00:40:30.055793 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:30.055640    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/49429502-4592-4b95-a41b-7f831eff48da-system-cni-dir\") pod \"multus-additional-cni-plugins-l97c7\" (UID: \"49429502-4592-4b95-a41b-7f831eff48da\") " pod="openshift-multus/multus-additional-cni-plugins-l97c7"
Jul 16 00:40:30.055793 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:30.055663    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/24b5fdac-6964-47c8-aa9e-d268d052c2e3-etc-sysconfig\") pod \"tuned-rjk6r\" (UID: \"24b5fdac-6964-47c8-aa9e-d268d052c2e3\") " pod="openshift-cluster-node-tuning-operator/tuned-rjk6r"
Jul 16 00:40:30.055793 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:30.055674    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e990a21d-614b-4d1d-be54-0df6e44487ed-log-socket\") pod \"ovnkube-node-bn5hr\" (UID: \"e990a21d-614b-4d1d-be54-0df6e44487ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn5hr"
Jul 16 00:40:30.055793 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:30.055687    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/24b5fdac-6964-47c8-aa9e-d268d052c2e3-run\") pod \"tuned-rjk6r\" (UID: \"24b5fdac-6964-47c8-aa9e-d268d052c2e3\") " pod="openshift-cluster-node-tuning-operator/tuned-rjk6r"
Jul 16 00:40:30.055793 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:30.055711    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/24b5fdac-6964-47c8-aa9e-d268d052c2e3-sys\") pod \"tuned-rjk6r\" (UID: \"24b5fdac-6964-47c8-aa9e-d268d052c2e3\") " pod="openshift-cluster-node-tuning-operator/tuned-rjk6r"
Jul 16 00:40:30.055793 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:30.055714    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/49429502-4592-4b95-a41b-7f831eff48da-system-cni-dir\") pod \"multus-additional-cni-plugins-l97c7\" (UID: \"49429502-4592-4b95-a41b-7f831eff48da\") " pod="openshift-multus/multus-additional-cni-plugins-l97c7"
Jul 16 00:40:30.055793 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:30.055740    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/16a7fa19-0ce5-49ca-96ef-dde5ae694dc0-kubelet-config\") pod \"global-pull-secret-syncer-hgcmx\" (UID: \"16a7fa19-0ce5-49ca-96ef-dde5ae694dc0\") " pod="kube-system/global-pull-secret-syncer-hgcmx"
Jul 16 00:40:30.055793 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:30.055785    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/16a7fa19-0ce5-49ca-96ef-dde5ae694dc0-kubelet-config\") pod \"global-pull-secret-syncer-hgcmx\" (UID: \"16a7fa19-0ce5-49ca-96ef-dde5ae694dc0\") " pod="kube-system/global-pull-secret-syncer-hgcmx"
Jul 16 00:40:30.056673 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:30.055787    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvkbs\" (UniqueName: \"kubernetes.io/projected/bd81358e-a649-4ba2-b63e-d9ba85ee93db-kube-api-access-lvkbs\") pod \"node-resolver-gzbcl\" (UID: \"bd81358e-a649-4ba2-b63e-d9ba85ee93db\") " pod="openshift-dns/node-resolver-gzbcl"
Jul 16 00:40:30.056673 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:30.055742    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/24b5fdac-6964-47c8-aa9e-d268d052c2e3-etc-sysconfig\") pod \"tuned-rjk6r\" (UID: \"24b5fdac-6964-47c8-aa9e-d268d052c2e3\") " pod="openshift-cluster-node-tuning-operator/tuned-rjk6r"
Jul 16 00:40:30.056673 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:30.055787    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/24b5fdac-6964-47c8-aa9e-d268d052c2e3-run\") pod \"tuned-rjk6r\" (UID: \"24b5fdac-6964-47c8-aa9e-d268d052c2e3\") " pod="openshift-cluster-node-tuning-operator/tuned-rjk6r"
Jul 16 00:40:30.056673 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:30.055822    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e990a21d-614b-4d1d-be54-0df6e44487ed-ovnkube-script-lib\") pod \"ovnkube-node-bn5hr\" (UID: \"e990a21d-614b-4d1d-be54-0df6e44487ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn5hr"
Jul 16 00:40:30.056673 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:30.055858    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/24b5fdac-6964-47c8-aa9e-d268d052c2e3-etc-sysctl-conf\") pod \"tuned-rjk6r\" (UID: \"24b5fdac-6964-47c8-aa9e-d268d052c2e3\") " pod="openshift-cluster-node-tuning-operator/tuned-rjk6r"
Jul 16 00:40:30.056673 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:30.055833    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/24b5fdac-6964-47c8-aa9e-d268d052c2e3-sys\") pod \"tuned-rjk6r\" (UID: \"24b5fdac-6964-47c8-aa9e-d268d052c2e3\") " pod="openshift-cluster-node-tuning-operator/tuned-rjk6r"
Jul 16 00:40:30.056673 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:30.055893    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e990a21d-614b-4d1d-be54-0df6e44487ed-ovnkube-config\") pod \"ovnkube-node-bn5hr\" (UID: \"e990a21d-614b-4d1d-be54-0df6e44487ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn5hr"
Jul 16 00:40:30.056673 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:30.055954    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/24b5fdac-6964-47c8-aa9e-d268d052c2e3-etc-systemd\") pod \"tuned-rjk6r\" (UID: \"24b5fdac-6964-47c8-aa9e-d268d052c2e3\") " pod="openshift-cluster-node-tuning-operator/tuned-rjk6r"
Jul 16 00:40:30.056673 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:30.055979    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/24b5fdac-6964-47c8-aa9e-d268d052c2e3-tmp\") pod \"tuned-rjk6r\" (UID: \"24b5fdac-6964-47c8-aa9e-d268d052c2e3\") " pod="openshift-cluster-node-tuning-operator/tuned-rjk6r"
Jul 16 00:40:30.056673 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:30.055951    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/39f3a671-1715-4dd3-9ca0-0b714dde3281-konnectivity-ca\") pod \"konnectivity-agent-tm5f5\" (UID: \"39f3a671-1715-4dd3-9ca0-0b714dde3281\") " pod="kube-system/konnectivity-agent-tm5f5"
Jul 16 00:40:30.056673 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:30.056019    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/24b5fdac-6964-47c8-aa9e-d268d052c2e3-etc-systemd\") pod \"tuned-rjk6r\" (UID: \"24b5fdac-6964-47c8-aa9e-d268d052c2e3\") " pod="openshift-cluster-node-tuning-operator/tuned-rjk6r"
Jul 16 00:40:30.056673 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:30.056004    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/49429502-4592-4b95-a41b-7f831eff48da-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-l97c7\" (UID: \"49429502-4592-4b95-a41b-7f831eff48da\") " pod="openshift-multus/multus-additional-cni-plugins-l97c7"
Jul 16 00:40:30.056673 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:30.056068    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/24b5fdac-6964-47c8-aa9e-d268d052c2e3-etc-sysctl-conf\") pod \"tuned-rjk6r\" (UID: \"24b5fdac-6964-47c8-aa9e-d268d052c2e3\") " pod="openshift-cluster-node-tuning-operator/tuned-rjk6r"
Jul 16 00:40:30.056673 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:30.056077    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/16a7fa19-0ce5-49ca-96ef-dde5ae694dc0-dbus\") pod \"global-pull-secret-syncer-hgcmx\" (UID: \"16a7fa19-0ce5-49ca-96ef-dde5ae694dc0\") " pod="kube-system/global-pull-secret-syncer-hgcmx"
Jul 16 00:40:30.056673 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:30.056124    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e990a21d-614b-4d1d-be54-0df6e44487ed-host-kubelet\") pod \"ovnkube-node-bn5hr\" (UID: \"e990a21d-614b-4d1d-be54-0df6e44487ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn5hr"
Jul 16 00:40:30.056673 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:30.056153    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e990a21d-614b-4d1d-be54-0df6e44487ed-var-lib-openvswitch\") pod \"ovnkube-node-bn5hr\" (UID: \"e990a21d-614b-4d1d-be54-0df6e44487ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn5hr"
Jul 16 00:40:30.056673 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:30.056182    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e990a21d-614b-4d1d-be54-0df6e44487ed-run-ovn\") pod \"ovnkube-node-bn5hr\" (UID: \"e990a21d-614b-4d1d-be54-0df6e44487ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn5hr"
Jul 16 00:40:30.056673 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:30.056189    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/16a7fa19-0ce5-49ca-96ef-dde5ae694dc0-dbus\") pod \"global-pull-secret-syncer-hgcmx\" (UID: \"16a7fa19-0ce5-49ca-96ef-dde5ae694dc0\") " pod="kube-system/global-pull-secret-syncer-hgcmx"
Jul 16 00:40:30.057611 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:30.056207    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e990a21d-614b-4d1d-be54-0df6e44487ed-host-cni-bin\") pod \"ovnkube-node-bn5hr\" (UID: \"e990a21d-614b-4d1d-be54-0df6e44487ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn5hr"
Jul 16 00:40:30.057611 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:30.056233    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/49429502-4592-4b95-a41b-7f831eff48da-cnibin\") pod \"multus-additional-cni-plugins-l97c7\" (UID: \"49429502-4592-4b95-a41b-7f831eff48da\") " pod="openshift-multus/multus-additional-cni-plugins-l97c7"
Jul 16 00:40:30.057611 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:30.056236    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e990a21d-614b-4d1d-be54-0df6e44487ed-var-lib-openvswitch\") pod \"ovnkube-node-bn5hr\" (UID: \"e990a21d-614b-4d1d-be54-0df6e44487ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn5hr"
Jul 16 00:40:30.057611 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:30.056259    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5a58757b-87fa-4446-b066-015fb48797cf-sys-fs\") pod \"aws-ebs-csi-driver-node-sfpw7\" (UID: \"5a58757b-87fa-4446-b066-015fb48797cf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sfpw7"
Jul 16 00:40:30.057611 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:30.056274    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e990a21d-614b-4d1d-be54-0df6e44487ed-host-kubelet\") pod \"ovnkube-node-bn5hr\" (UID: \"e990a21d-614b-4d1d-be54-0df6e44487ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn5hr"
Jul 16 00:40:30.057611 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:30.056285    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/24b5fdac-6964-47c8-aa9e-d268d052c2e3-etc-modprobe-d\") pod \"tuned-rjk6r\" (UID: \"24b5fdac-6964-47c8-aa9e-d268d052c2e3\") " pod="openshift-cluster-node-tuning-operator/tuned-rjk6r"
Jul 16 00:40:30.057611 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:30.056313    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrq8r\" (UniqueName: \"kubernetes.io/projected/24b5fdac-6964-47c8-aa9e-d268d052c2e3-kube-api-access-jrq8r\") pod \"tuned-rjk6r\" (UID: \"24b5fdac-6964-47c8-aa9e-d268d052c2e3\") " pod="openshift-cluster-node-tuning-operator/tuned-rjk6r"
Jul 16 00:40:30.057611 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:30.056329    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e990a21d-614b-4d1d-be54-0df6e44487ed-run-ovn\") pod \"ovnkube-node-bn5hr\" (UID: \"e990a21d-614b-4d1d-be54-0df6e44487ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn5hr"
Jul 16 00:40:30.057611 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:30.056342    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e990a21d-614b-4d1d-be54-0df6e44487ed-ovn-node-metrics-cert\") pod \"ovnkube-node-bn5hr\" (UID: \"e990a21d-614b-4d1d-be54-0df6e44487ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn5hr"
Jul 16 00:40:30.057611 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:30.056369    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/49429502-4592-4b95-a41b-7f831eff48da-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-l97c7\" (UID: \"49429502-4592-4b95-a41b-7f831eff48da\") " pod="openshift-multus/multus-additional-cni-plugins-l97c7"
Jul 16 00:40:30.057611 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:30.056395    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/24b5fdac-6964-47c8-aa9e-d268d052c2e3-etc-kubernetes\") pod \"tuned-rjk6r\" (UID: \"24b5fdac-6964-47c8-aa9e-d268d052c2e3\") " pod="openshift-cluster-node-tuning-operator/tuned-rjk6r"
Jul 16 00:40:30.057611 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:30.056421    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/bd81358e-a649-4ba2-b63e-d9ba85ee93db-hosts-file\") pod \"node-resolver-gzbcl\" (UID: \"bd81358e-a649-4ba2-b63e-d9ba85ee93db\") " pod="openshift-dns/node-resolver-gzbcl"
Jul 16 00:40:30.057611 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:30.056424    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/24b5fdac-6964-47c8-aa9e-d268d052c2e3-etc-modprobe-d\") pod \"tuned-rjk6r\" (UID: \"24b5fdac-6964-47c8-aa9e-d268d052c2e3\") " pod="openshift-cluster-node-tuning-operator/tuned-rjk6r"
Jul 16 00:40:30.057611 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:30.056458    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e990a21d-614b-4d1d-be54-0df6e44487ed-host-cni-netd\") pod \"ovnkube-node-bn5hr\" (UID: \"e990a21d-614b-4d1d-be54-0df6e44487ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn5hr"
Jul 16 00:40:30.057611 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:30.056485    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5a58757b-87fa-4446-b066-015fb48797cf-sys-fs\") pod \"aws-ebs-csi-driver-node-sfpw7\" (UID: \"5a58757b-87fa-4446-b066-015fb48797cf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sfpw7"
Jul 16 00:40:30.057611 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:30.056485    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/39f3a671-1715-4dd3-9ca0-0b714dde3281-agent-certs\") pod \"konnectivity-agent-tm5f5\" (UID: \"39f3a671-1715-4dd3-9ca0-0b714dde3281\") " pod="kube-system/konnectivity-agent-tm5f5"
Jul 16 00:40:30.057611 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:30.056496    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e990a21d-614b-4d1d-be54-0df6e44487ed-ovnkube-script-lib\") pod \"ovnkube-node-bn5hr\" (UID: \"e990a21d-614b-4d1d-be54-0df6e44487ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn5hr"
Jul 16 00:40:30.058497 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:30.056526    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5a58757b-87fa-4446-b066-015fb48797cf-registration-dir\") pod \"aws-ebs-csi-driver-node-sfpw7\" (UID: \"5a58757b-87fa-4446-b066-015fb48797cf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sfpw7"
Jul 16 00:40:30.058497 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:30.056545    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/49429502-4592-4b95-a41b-7f831eff48da-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-l97c7\" (UID: \"49429502-4592-4b95-a41b-7f831eff48da\") " pod="openshift-multus/multus-additional-cni-plugins-l97c7"
Jul 16 00:40:30.058497 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:30.056593    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5a58757b-87fa-4446-b066-015fb48797cf-registration-dir\") pod \"aws-ebs-csi-driver-node-sfpw7\" (UID: \"5a58757b-87fa-4446-b066-015fb48797cf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sfpw7"
Jul 16 00:40:30.058497 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:30.056620    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e990a21d-614b-4d1d-be54-0df6e44487ed-host-cni-bin\") pod \"ovnkube-node-bn5hr\" (UID: \"e990a21d-614b-4d1d-be54-0df6e44487ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn5hr"
Jul 16 00:40:30.058497 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:30.056865    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/bd81358e-a649-4ba2-b63e-d9ba85ee93db-hosts-file\") pod \"node-resolver-gzbcl\" (UID: \"bd81358e-a649-4ba2-b63e-d9ba85ee93db\") " pod="openshift-dns/node-resolver-gzbcl"
Jul 16 00:40:30.058497 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:30.056916    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/24b5fdac-6964-47c8-aa9e-d268d052c2e3-etc-kubernetes\") pod \"tuned-rjk6r\" (UID: \"24b5fdac-6964-47c8-aa9e-d268d052c2e3\") " pod="openshift-cluster-node-tuning-operator/tuned-rjk6r"
Jul 16 00:40:30.058497 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:30.056935    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/49429502-4592-4b95-a41b-7f831eff48da-cnibin\") pod \"multus-additional-cni-plugins-l97c7\" (UID: \"49429502-4592-4b95-a41b-7f831eff48da\") " pod="openshift-multus/multus-additional-cni-plugins-l97c7"
Jul 16 00:40:30.058497 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:30.056969    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e990a21d-614b-4d1d-be54-0df6e44487ed-host-cni-netd\") pod \"ovnkube-node-bn5hr\" (UID: \"e990a21d-614b-4d1d-be54-0df6e44487ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn5hr"
Jul 16 00:40:30.058497 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:30.057000    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e990a21d-614b-4d1d-be54-0df6e44487ed-run-systemd\") pod \"ovnkube-node-bn5hr\" (UID: \"e990a21d-614b-4d1d-be54-0df6e44487ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn5hr"
Jul 16 00:40:30.058497 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:30.057057    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/49429502-4592-4b95-a41b-7f831eff48da-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-l97c7\" (UID: \"49429502-4592-4b95-a41b-7f831eff48da\") " pod="openshift-multus/multus-additional-cni-plugins-l97c7"
Jul 16 00:40:30.058497 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:30.057120    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e990a21d-614b-4d1d-be54-0df6e44487ed-run-openvswitch\") pod \"ovnkube-node-bn5hr\" (UID: \"e990a21d-614b-4d1d-be54-0df6e44487ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn5hr"
Jul 16 00:40:30.058497 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:30.057150    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/49429502-4592-4b95-a41b-7f831eff48da-tuning-conf-dir\") pod \"multus-additional-cni-plugins-l97c7\" (UID: \"49429502-4592-4b95-a41b-7f831eff48da\") " pod="openshift-multus/multus-additional-cni-plugins-l97c7"
Jul 16 00:40:30.058497 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:30.057194    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/051e94d1-76fa-484b-b118-b29951f4c5ae-metrics-certs\") pod \"network-metrics-daemon-bkf2h\" (UID: \"051e94d1-76fa-484b-b118-b29951f4c5ae\") " pod="openshift-multus/network-metrics-daemon-bkf2h"
Jul 16 00:40:30.058497 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:30.057221    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvtjv\" (UniqueName: \"kubernetes.io/projected/051e94d1-76fa-484b-b118-b29951f4c5ae-kube-api-access-wvtjv\") pod \"network-metrics-daemon-bkf2h\" (UID: \"051e94d1-76fa-484b-b118-b29951f4c5ae\") " pod="openshift-multus/network-metrics-daemon-bkf2h"
Jul 16 00:40:30.058497 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:30.057248    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/24b5fdac-6964-47c8-aa9e-d268d052c2e3-var-lib-kubelet\") pod \"tuned-rjk6r\" (UID: \"24b5fdac-6964-47c8-aa9e-d268d052c2e3\") " pod="openshift-cluster-node-tuning-operator/tuned-rjk6r"
Jul 16 00:40:30.058497 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:30.057263    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e990a21d-614b-4d1d-be54-0df6e44487ed-run-systemd\") pod \"ovnkube-node-bn5hr\" (UID: \"e990a21d-614b-4d1d-be54-0df6e44487ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn5hr"
Jul 16 00:40:30.058497 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:30.057309    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f0b6d16f-9a13-480c-b981-6481983dbd91-serviceca\") pod \"node-ca-rk85k\" (UID: \"f0b6d16f-9a13-480c-b981-6481983dbd91\") " pod="openshift-image-registry/node-ca-rk85k"
Jul 16 00:40:30.059395 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:30.057338    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e990a21d-614b-4d1d-be54-0df6e44487ed-host-slash\") pod \"ovnkube-node-bn5hr\" (UID: \"e990a21d-614b-4d1d-be54-0df6e44487ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn5hr"
Jul 16 00:40:30.059395 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:40:30.057364    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 00:40:30.059395 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:30.057373    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e990a21d-614b-4d1d-be54-0df6e44487ed-node-log\") pod \"ovnkube-node-bn5hr\" (UID: \"e990a21d-614b-4d1d-be54-0df6e44487ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn5hr"
Jul 16 00:40:30.059395 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:30.057432    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/24b5fdac-6964-47c8-aa9e-d268d052c2e3-var-lib-kubelet\") pod \"tuned-rjk6r\" (UID: \"24b5fdac-6964-47c8-aa9e-d268d052c2e3\") " pod="openshift-cluster-node-tuning-operator/tuned-rjk6r"
Jul 16 00:40:30.059395 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:40:30.057447    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/051e94d1-76fa-484b-b118-b29951f4c5ae-metrics-certs podName:051e94d1-76fa-484b-b118-b29951f4c5ae nodeName:}" failed. No retries permitted until 2026-07-16 00:40:30.557428491 +0000 UTC m=+2.992690231 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/051e94d1-76fa-484b-b118-b29951f4c5ae-metrics-certs") pod "network-metrics-daemon-bkf2h" (UID: "051e94d1-76fa-484b-b118-b29951f4c5ae") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 00:40:30.059395 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:30.057465    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e990a21d-614b-4d1d-be54-0df6e44487ed-run-openvswitch\") pod \"ovnkube-node-bn5hr\" (UID: \"e990a21d-614b-4d1d-be54-0df6e44487ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn5hr"
Jul 16 00:40:30.059395 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:30.057495    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5a58757b-87fa-4446-b066-015fb48797cf-socket-dir\") pod \"aws-ebs-csi-driver-node-sfpw7\" (UID: \"5a58757b-87fa-4446-b066-015fb48797cf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sfpw7"
Jul 16 00:40:30.059395 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:30.057522    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e990a21d-614b-4d1d-be54-0df6e44487ed-node-log\") pod \"ovnkube-node-bn5hr\" (UID: \"e990a21d-614b-4d1d-be54-0df6e44487ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn5hr"
Jul 16 00:40:30.059395 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:30.057560    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5p2f\" (UniqueName: \"kubernetes.io/projected/e990a21d-614b-4d1d-be54-0df6e44487ed-kube-api-access-x5p2f\") pod \"ovnkube-node-bn5hr\" (UID: \"e990a21d-614b-4d1d-be54-0df6e44487ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn5hr"
Jul 16 00:40:30.059395 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:30.057602    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/49429502-4592-4b95-a41b-7f831eff48da-tuning-conf-dir\") pod \"multus-additional-cni-plugins-l97c7\" (UID: \"49429502-4592-4b95-a41b-7f831eff48da\") " pod="openshift-multus/multus-additional-cni-plugins-l97c7"
Jul 16 00:40:30.059395 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:30.057610    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcsrk\" (UniqueName: \"kubernetes.io/projected/49429502-4592-4b95-a41b-7f831eff48da-kube-api-access-jcsrk\") pod \"multus-additional-cni-plugins-l97c7\" (UID: \"49429502-4592-4b95-a41b-7f831eff48da\") " pod="openshift-multus/multus-additional-cni-plugins-l97c7"
Jul 16 00:40:30.059395 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:30.057655    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e990a21d-614b-4d1d-be54-0df6e44487ed-host-slash\") pod \"ovnkube-node-bn5hr\" (UID: \"e990a21d-614b-4d1d-be54-0df6e44487ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn5hr"
Jul 16 00:40:30.059395 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:30.057715    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5a58757b-87fa-4446-b066-015fb48797cf-kubelet-dir\") pod \"aws-ebs-csi-driver-node-sfpw7\" (UID: \"5a58757b-87fa-4446-b066-015fb48797cf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sfpw7"
Jul 16 00:40:30.059395 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:30.057783    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5a58757b-87fa-4446-b066-015fb48797cf-device-dir\") pod \"aws-ebs-csi-driver-node-sfpw7\" (UID: \"5a58757b-87fa-4446-b066-015fb48797cf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sfpw7"
Jul 16 00:40:30.059395 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:30.057827    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5a58757b-87fa-4446-b066-015fb48797cf-etc-selinux\") pod \"aws-ebs-csi-driver-node-sfpw7\" (UID: \"5a58757b-87fa-4446-b066-015fb48797cf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sfpw7"
Jul 16 00:40:30.059395 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:30.057899    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5a58757b-87fa-4446-b066-015fb48797cf-kubelet-dir\") pod \"aws-ebs-csi-driver-node-sfpw7\" (UID: \"5a58757b-87fa-4446-b066-015fb48797cf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sfpw7"
Jul 16 00:40:30.059395 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:30.057962    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5a58757b-87fa-4446-b066-015fb48797cf-socket-dir\") pod \"aws-ebs-csi-driver-node-sfpw7\" (UID: \"5a58757b-87fa-4446-b066-015fb48797cf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sfpw7"
Jul 16 00:40:30.060313 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:30.058012    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5a58757b-87fa-4446-b066-015fb48797cf-etc-selinux\") pod \"aws-ebs-csi-driver-node-sfpw7\" (UID: \"5a58757b-87fa-4446-b066-015fb48797cf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sfpw7"
Jul 16 00:40:30.060313 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:30.058021    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/24b5fdac-6964-47c8-aa9e-d268d052c2e3-lib-modules\") pod \"tuned-rjk6r\" (UID: \"24b5fdac-6964-47c8-aa9e-d268d052c2e3\") " pod="openshift-cluster-node-tuning-operator/tuned-rjk6r"
Jul 16 00:40:30.060313 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:30.058055    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5a58757b-87fa-4446-b066-015fb48797cf-device-dir\") pod \"aws-ebs-csi-driver-node-sfpw7\" (UID: \"5a58757b-87fa-4446-b066-015fb48797cf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sfpw7"
Jul 16 00:40:30.060313 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:30.058103    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/24b5fdac-6964-47c8-aa9e-d268d052c2e3-host\") pod \"tuned-rjk6r\" (UID: \"24b5fdac-6964-47c8-aa9e-d268d052c2e3\") " pod="openshift-cluster-node-tuning-operator/tuned-rjk6r"
Jul 16 00:40:30.060313 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:30.058131    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd81358e-a649-4ba2-b63e-d9ba85ee93db-tmp-dir\") pod \"node-resolver-gzbcl\" (UID: \"bd81358e-a649-4ba2-b63e-d9ba85ee93db\") " pod="openshift-dns/node-resolver-gzbcl"
Jul 16 00:40:30.060313 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:30.058175    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/49429502-4592-4b95-a41b-7f831eff48da-cni-binary-copy\") pod \"multus-additional-cni-plugins-l97c7\" (UID: \"49429502-4592-4b95-a41b-7f831eff48da\") " pod="openshift-multus/multus-additional-cni-plugins-l97c7"
Jul 16 00:40:30.060313 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:30.058232    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/24b5fdac-6964-47c8-aa9e-d268d052c2e3-host\") pod \"tuned-rjk6r\" (UID: \"24b5fdac-6964-47c8-aa9e-d268d052c2e3\") " pod="openshift-cluster-node-tuning-operator/tuned-rjk6r"
Jul 16 00:40:30.060313 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:30.058290    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f0b6d16f-9a13-480c-b981-6481983dbd91-serviceca\") pod \"node-ca-rk85k\" (UID: \"f0b6d16f-9a13-480c-b981-6481983dbd91\") " pod="openshift-image-registry/node-ca-rk85k"
Jul 16 00:40:30.060313 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:30.058477    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e990a21d-614b-4d1d-be54-0df6e44487ed-ovnkube-config\") pod \"ovnkube-node-bn5hr\" (UID: \"e990a21d-614b-4d1d-be54-0df6e44487ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn5hr"
Jul 16 00:40:30.060313 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:30.058629    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd81358e-a649-4ba2-b63e-d9ba85ee93db-tmp-dir\") pod \"node-resolver-gzbcl\" (UID: \"bd81358e-a649-4ba2-b63e-d9ba85ee93db\") " pod="openshift-dns/node-resolver-gzbcl"
Jul 16 00:40:30.060313 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:30.058634    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/24b5fdac-6964-47c8-aa9e-d268d052c2e3-tmp\") pod \"tuned-rjk6r\" (UID: \"24b5fdac-6964-47c8-aa9e-d268d052c2e3\") " pod="openshift-cluster-node-tuning-operator/tuned-rjk6r"
Jul 16 00:40:30.060313 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:30.058674    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/24b5fdac-6964-47c8-aa9e-d268d052c2e3-etc-tuned\") pod \"tuned-rjk6r\" (UID: \"24b5fdac-6964-47c8-aa9e-d268d052c2e3\") " pod="openshift-cluster-node-tuning-operator/tuned-rjk6r"
Jul 16 00:40:30.060313 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:30.058705    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/49429502-4592-4b95-a41b-7f831eff48da-cni-binary-copy\") pod \"multus-additional-cni-plugins-l97c7\" (UID: \"49429502-4592-4b95-a41b-7f831eff48da\") " pod="openshift-multus/multus-additional-cni-plugins-l97c7"
Jul 16 00:40:30.060313 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:30.058714    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e990a21d-614b-4d1d-be54-0df6e44487ed-env-overrides\") pod \"ovnkube-node-bn5hr\" (UID: \"e990a21d-614b-4d1d-be54-0df6e44487ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn5hr"
Jul 16 00:40:30.060313 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:30.058751    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f0b6d16f-9a13-480c-b981-6481983dbd91-host\") pod \"node-ca-rk85k\" (UID: \"f0b6d16f-9a13-480c-b981-6481983dbd91\") " pod="openshift-image-registry/node-ca-rk85k"
Jul 16 00:40:30.060313 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:30.058781    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e990a21d-614b-4d1d-be54-0df6e44487ed-systemd-units\") pod \"ovnkube-node-bn5hr\" (UID: \"e990a21d-614b-4d1d-be54-0df6e44487ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn5hr"
Jul 16 00:40:30.060313 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:30.058818    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e990a21d-614b-4d1d-be54-0df6e44487ed-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bn5hr\" (UID: \"e990a21d-614b-4d1d-be54-0df6e44487ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn5hr"
Jul 16 00:40:30.060313 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:30.058849    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqw59\" (UniqueName: \"kubernetes.io/projected/f0b6d16f-9a13-480c-b981-6481983dbd91-kube-api-access-xqw59\") pod \"node-ca-rk85k\" (UID: \"f0b6d16f-9a13-480c-b981-6481983dbd91\") " pod="openshift-image-registry/node-ca-rk85k"
Jul 16 00:40:30.061296 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:30.058923    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e990a21d-614b-4d1d-be54-0df6e44487ed-systemd-units\") pod \"ovnkube-node-bn5hr\" (UID: \"e990a21d-614b-4d1d-be54-0df6e44487ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn5hr"
Jul 16 00:40:30.061296 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:30.058976    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f0b6d16f-9a13-480c-b981-6481983dbd91-host\") pod \"node-ca-rk85k\" (UID: \"f0b6d16f-9a13-480c-b981-6481983dbd91\") " pod="openshift-image-registry/node-ca-rk85k"
Jul 16 00:40:30.061296 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:30.059034    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e990a21d-614b-4d1d-be54-0df6e44487ed-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bn5hr\" (UID: \"e990a21d-614b-4d1d-be54-0df6e44487ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn5hr"
Jul 16 00:40:30.061296 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:30.059117    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e990a21d-614b-4d1d-be54-0df6e44487ed-env-overrides\") pod \"ovnkube-node-bn5hr\" (UID: \"e990a21d-614b-4d1d-be54-0df6e44487ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn5hr"
Jul 16 00:40:30.061296 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:30.059307    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/24b5fdac-6964-47c8-aa9e-d268d052c2e3-lib-modules\") pod \"tuned-rjk6r\" (UID: \"24b5fdac-6964-47c8-aa9e-d268d052c2e3\") " pod="openshift-cluster-node-tuning-operator/tuned-rjk6r"
Jul 16 00:40:30.061296 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:30.060785    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/39f3a671-1715-4dd3-9ca0-0b714dde3281-agent-certs\") pod \"konnectivity-agent-tm5f5\" (UID: \"39f3a671-1715-4dd3-9ca0-0b714dde3281\") " pod="kube-system/konnectivity-agent-tm5f5"
Jul 16 00:40:30.061597 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:30.061329    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e990a21d-614b-4d1d-be54-0df6e44487ed-ovn-node-metrics-cert\") pod \"ovnkube-node-bn5hr\" (UID: \"e990a21d-614b-4d1d-be54-0df6e44487ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn5hr"
Jul 16 00:40:30.072647 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:40:30.072619    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 16 00:40:30.072647 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:40:30.072642    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 16 00:40:30.072647 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:40:30.072653    2563 projected.go:196] Error preparing data for projected volume kube-api-access-t7j7f for pod openshift-network-diagnostics/network-check-target-btw92: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 00:40:30.072866 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:40:30.072712    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1a9f4afb-53aa-401c-9a57-0cfc7e2f6af0-kube-api-access-t7j7f podName:1a9f4afb-53aa-401c-9a57-0cfc7e2f6af0 nodeName:}" failed. No retries permitted until 2026-07-16 00:40:30.572693321 +0000 UTC m=+3.007955057 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-t7j7f" (UniqueName: "kubernetes.io/projected/1a9f4afb-53aa-401c-9a57-0cfc7e2f6af0-kube-api-access-t7j7f") pod "network-check-target-btw92" (UID: "1a9f4afb-53aa-401c-9a57-0cfc7e2f6af0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 00:40:30.077450 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:30.077422    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7npff\" (UniqueName: \"kubernetes.io/projected/5a58757b-87fa-4446-b066-015fb48797cf-kube-api-access-7npff\") pod \"aws-ebs-csi-driver-node-sfpw7\" (UID: \"5a58757b-87fa-4446-b066-015fb48797cf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sfpw7"
Jul 16 00:40:30.077586 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:30.077480    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5p2f\" (UniqueName: \"kubernetes.io/projected/e990a21d-614b-4d1d-be54-0df6e44487ed-kube-api-access-x5p2f\") pod \"ovnkube-node-bn5hr\" (UID: \"e990a21d-614b-4d1d-be54-0df6e44487ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn5hr"
Jul 16 00:40:30.077758 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:30.077736    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcsrk\" (UniqueName: \"kubernetes.io/projected/49429502-4592-4b95-a41b-7f831eff48da-kube-api-access-jcsrk\") pod \"multus-additional-cni-plugins-l97c7\" (UID: \"49429502-4592-4b95-a41b-7f831eff48da\") " pod="openshift-multus/multus-additional-cni-plugins-l97c7"
Jul 16 00:40:30.077829 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:30.077801    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvkbs\" (UniqueName: \"kubernetes.io/projected/bd81358e-a649-4ba2-b63e-d9ba85ee93db-kube-api-access-lvkbs\") pod \"node-resolver-gzbcl\" (UID: \"bd81358e-a649-4ba2-b63e-d9ba85ee93db\") " pod="openshift-dns/node-resolver-gzbcl"
Jul 16 00:40:30.078195 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:30.078174    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvtjv\" (UniqueName: \"kubernetes.io/projected/051e94d1-76fa-484b-b118-b29951f4c5ae-kube-api-access-wvtjv\") pod \"network-metrics-daemon-bkf2h\" (UID: \"051e94d1-76fa-484b-b118-b29951f4c5ae\") " pod="openshift-multus/network-metrics-daemon-bkf2h"
Jul 16 00:40:30.078906 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:30.078884    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrq8r\" (UniqueName: \"kubernetes.io/projected/24b5fdac-6964-47c8-aa9e-d268d052c2e3-kube-api-access-jrq8r\") pod \"tuned-rjk6r\" (UID: \"24b5fdac-6964-47c8-aa9e-d268d052c2e3\") " pod="openshift-cluster-node-tuning-operator/tuned-rjk6r"
Jul 16 00:40:30.078999 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:30.078970    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqw59\" (UniqueName: \"kubernetes.io/projected/f0b6d16f-9a13-480c-b981-6481983dbd91-kube-api-access-xqw59\") pod \"node-ca-rk85k\" (UID: \"f0b6d16f-9a13-480c-b981-6481983dbd91\") " pod="openshift-image-registry/node-ca-rk85k"
Jul 16 00:40:30.153743 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:30.153712    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-h4ks8"
Jul 16 00:40:30.166200 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:30.166175    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sfpw7"
Jul 16 00:40:30.177146 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:30.177119    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-ksj4j"
Jul 16 00:40:30.183600 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:30.183580    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rjk6r"
Jul 16 00:40:30.192491 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:30.192470    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-gzbcl"
Jul 16 00:40:30.200373 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:30.200353    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-rk85k"
Jul 16 00:40:30.221976 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:30.221944    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-bn5hr"
Jul 16 00:40:30.229522 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:30.229505    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-l97c7"
Jul 16 00:40:30.231662 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:30.231643    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-tm5f5"
Jul 16 00:40:30.562646 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:30.562617    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/051e94d1-76fa-484b-b118-b29951f4c5ae-metrics-certs\") pod \"network-metrics-daemon-bkf2h\" (UID: \"051e94d1-76fa-484b-b118-b29951f4c5ae\") " pod="openshift-multus/network-metrics-daemon-bkf2h"
Jul 16 00:40:30.562837 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:30.562680    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/16a7fa19-0ce5-49ca-96ef-dde5ae694dc0-original-pull-secret\") pod \"global-pull-secret-syncer-hgcmx\" (UID: \"16a7fa19-0ce5-49ca-96ef-dde5ae694dc0\") " pod="kube-system/global-pull-secret-syncer-hgcmx"
Jul 16 00:40:30.562837 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:40:30.562792    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 00:40:30.562837 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:40:30.562823    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 16 00:40:30.562987 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:40:30.562886    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/051e94d1-76fa-484b-b118-b29951f4c5ae-metrics-certs podName:051e94d1-76fa-484b-b118-b29951f4c5ae nodeName:}" failed. No retries permitted until 2026-07-16 00:40:31.562851827 +0000 UTC m=+3.998113566 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/051e94d1-76fa-484b-b118-b29951f4c5ae-metrics-certs") pod "network-metrics-daemon-bkf2h" (UID: "051e94d1-76fa-484b-b118-b29951f4c5ae") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 00:40:30.562987 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:40:30.562907    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16a7fa19-0ce5-49ca-96ef-dde5ae694dc0-original-pull-secret podName:16a7fa19-0ce5-49ca-96ef-dde5ae694dc0 nodeName:}" failed. No retries permitted until 2026-07-16 00:40:31.562897168 +0000 UTC m=+3.998158892 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/16a7fa19-0ce5-49ca-96ef-dde5ae694dc0-original-pull-secret") pod "global-pull-secret-syncer-hgcmx" (UID: "16a7fa19-0ce5-49ca-96ef-dde5ae694dc0") : object "kube-system"/"original-pull-secret" not registered
Jul 16 00:40:30.663632 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:30.663601    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7j7f\" (UniqueName: \"kubernetes.io/projected/1a9f4afb-53aa-401c-9a57-0cfc7e2f6af0-kube-api-access-t7j7f\") pod \"network-check-target-btw92\" (UID: \"1a9f4afb-53aa-401c-9a57-0cfc7e2f6af0\") " pod="openshift-network-diagnostics/network-check-target-btw92"
Jul 16 00:40:30.663805 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:40:30.663738    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 16 00:40:30.663805 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:40:30.663759    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 16 00:40:30.663805 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:40:30.663770    2563 projected.go:196] Error preparing data for projected volume kube-api-access-t7j7f for pod openshift-network-diagnostics/network-check-target-btw92: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 00:40:30.663999 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:40:30.663823    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1a9f4afb-53aa-401c-9a57-0cfc7e2f6af0-kube-api-access-t7j7f podName:1a9f4afb-53aa-401c-9a57-0cfc7e2f6af0 nodeName:}" failed. No retries permitted until 2026-07-16 00:40:31.663811372 +0000 UTC m=+4.099073089 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-t7j7f" (UniqueName: "kubernetes.io/projected/1a9f4afb-53aa-401c-9a57-0cfc7e2f6af0-kube-api-access-t7j7f") pod "network-check-target-btw92" (UID: "1a9f4afb-53aa-401c-9a57-0cfc7e2f6af0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 00:40:30.954045 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:30.953935    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-15 00:35:29 +0000 UTC" deadline="2028-02-28 21:23:42.967245416 +0000 UTC"
Jul 16 00:40:30.954045 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:30.953993    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14228h43m12.013255665s"
Jul 16 00:40:31.076814 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:31.076787    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd81358e_a649_4ba2_b63e_d9ba85ee93db.slice/crio-6c548af5b532ce36d13b4ef1112d8670b48c1c1053a336defbfe04cf24ea6c28 WatchSource:0}: Error finding container 6c548af5b532ce36d13b4ef1112d8670b48c1c1053a336defbfe04cf24ea6c28: Status 404 returned error can't find the container with id 6c548af5b532ce36d13b4ef1112d8670b48c1c1053a336defbfe04cf24ea6c28
Jul 16 00:40:31.085584 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:31.085555    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a58757b_87fa_4446_b066_015fb48797cf.slice/crio-1c75f961bbcc617ff29626dcb8119e6cd9772778ba80226bd5abb402c7c7605e WatchSource:0}: Error finding container 1c75f961bbcc617ff29626dcb8119e6cd9772778ba80226bd5abb402c7c7605e: Status 404 returned error can't find the container with id 1c75f961bbcc617ff29626dcb8119e6cd9772778ba80226bd5abb402c7c7605e
Jul 16 00:40:31.087702 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:31.087670    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode990a21d_614b_4d1d_be54_0df6e44487ed.slice/crio-6e64da4e2977f0bab815abe75af84bbdcf525aeeef83234717b7393174989b35 WatchSource:0}: Error finding container 6e64da4e2977f0bab815abe75af84bbdcf525aeeef83234717b7393174989b35: Status 404 returned error can't find the container with id 6e64da4e2977f0bab815abe75af84bbdcf525aeeef83234717b7393174989b35
Jul 16 00:40:31.088563 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:31.088541    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded5df646_0604_408b_bd6e_a502f0bad8e1.slice/crio-d750c60ee44ccf6714dc10b8d5c4dda15ac69dd813ecc3dbd6e8dbba64355120 WatchSource:0}: Error finding container d750c60ee44ccf6714dc10b8d5c4dda15ac69dd813ecc3dbd6e8dbba64355120: Status 404 returned error can't find the container with id d750c60ee44ccf6714dc10b8d5c4dda15ac69dd813ecc3dbd6e8dbba64355120
Jul 16 00:40:31.090496 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:31.090471    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49429502_4592_4b95_a41b_7f831eff48da.slice/crio-9e7ce299e60abfcdb1252e0df40267644a1f98bfebacbb46db2f2e0fc4e159d2 WatchSource:0}: Error finding container 9e7ce299e60abfcdb1252e0df40267644a1f98bfebacbb46db2f2e0fc4e159d2: Status 404 returned error can't find the container with id 9e7ce299e60abfcdb1252e0df40267644a1f98bfebacbb46db2f2e0fc4e159d2
Jul 16 00:40:31.091772 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:31.091751    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0b6d16f_9a13_480c_b981_6481983dbd91.slice/crio-9fca5c9df05eba226d64326f5ff2fb7fc03872e79c1f119e0d06c22e25978561 WatchSource:0}: Error finding container 9fca5c9df05eba226d64326f5ff2fb7fc03872e79c1f119e0d06c22e25978561: Status 404 returned error can't find the container with id 9fca5c9df05eba226d64326f5ff2fb7fc03872e79c1f119e0d06c22e25978561
Jul 16 00:40:31.092725 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:31.092701    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod090eddb9_3f4c_4a94_9b62_4f6bf8f38428.slice/crio-fc1ad67418b847df1eecd1a6d9f98b8af023c6e9c17bc8cc0ae685f63f1881ca WatchSource:0}: Error finding container fc1ad67418b847df1eecd1a6d9f98b8af023c6e9c17bc8cc0ae685f63f1881ca: Status 404 returned error can't find the container with id fc1ad67418b847df1eecd1a6d9f98b8af023c6e9c17bc8cc0ae685f63f1881ca
Jul 16 00:40:31.094295 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:31.094263    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24b5fdac_6964_47c8_aa9e_d268d052c2e3.slice/crio-0efea6001e9f1f1e7ec997884b1bd43a20833ae3052d0327f82377d9177b5be8 WatchSource:0}: Error finding container 0efea6001e9f1f1e7ec997884b1bd43a20833ae3052d0327f82377d9177b5be8: Status 404 returned error can't find the container with id 0efea6001e9f1f1e7ec997884b1bd43a20833ae3052d0327f82377d9177b5be8
Jul 16 00:40:31.095883 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:40:31.095843    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39f3a671_1715_4dd3_9ca0_0b714dde3281.slice/crio-d68c55a527029ea4888ccc50e1496eb503e67bb29376a7ac1e3bcdff480b46bc WatchSource:0}: Error finding container d68c55a527029ea4888ccc50e1496eb503e67bb29376a7ac1e3bcdff480b46bc: Status 404 returned error can't find the container with id d68c55a527029ea4888ccc50e1496eb503e67bb29376a7ac1e3bcdff480b46bc
Jul 16 00:40:31.570333 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:31.570205    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/051e94d1-76fa-484b-b118-b29951f4c5ae-metrics-certs\") pod \"network-metrics-daemon-bkf2h\" (UID: \"051e94d1-76fa-484b-b118-b29951f4c5ae\") " pod="openshift-multus/network-metrics-daemon-bkf2h"
Jul 16 00:40:31.570333 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:31.570276    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/16a7fa19-0ce5-49ca-96ef-dde5ae694dc0-original-pull-secret\") pod \"global-pull-secret-syncer-hgcmx\" (UID: \"16a7fa19-0ce5-49ca-96ef-dde5ae694dc0\") " pod="kube-system/global-pull-secret-syncer-hgcmx"
Jul 16 00:40:31.570598 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:40:31.570414    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 16 00:40:31.570598 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:40:31.570477    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16a7fa19-0ce5-49ca-96ef-dde5ae694dc0-original-pull-secret podName:16a7fa19-0ce5-49ca-96ef-dde5ae694dc0 nodeName:}" failed. No retries permitted until 2026-07-16 00:40:33.570460156 +0000 UTC m=+6.005721880 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/16a7fa19-0ce5-49ca-96ef-dde5ae694dc0-original-pull-secret") pod "global-pull-secret-syncer-hgcmx" (UID: "16a7fa19-0ce5-49ca-96ef-dde5ae694dc0") : object "kube-system"/"original-pull-secret" not registered
Jul 16 00:40:31.570879 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:40:31.570850    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 00:40:31.570964 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:40:31.570916    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/051e94d1-76fa-484b-b118-b29951f4c5ae-metrics-certs podName:051e94d1-76fa-484b-b118-b29951f4c5ae nodeName:}" failed. No retries permitted until 2026-07-16 00:40:33.570900326 +0000 UTC m=+6.006162059 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/051e94d1-76fa-484b-b118-b29951f4c5ae-metrics-certs") pod "network-metrics-daemon-bkf2h" (UID: "051e94d1-76fa-484b-b118-b29951f4c5ae") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 00:40:31.671686 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:31.671599    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7j7f\" (UniqueName: \"kubernetes.io/projected/1a9f4afb-53aa-401c-9a57-0cfc7e2f6af0-kube-api-access-t7j7f\") pod \"network-check-target-btw92\" (UID: \"1a9f4afb-53aa-401c-9a57-0cfc7e2f6af0\") " pod="openshift-network-diagnostics/network-check-target-btw92"
Jul 16 00:40:31.671889 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:40:31.671855    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 16 00:40:31.671965 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:40:31.671897    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 16 00:40:31.671965 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:40:31.671910    2563 projected.go:196] Error preparing data for projected volume kube-api-access-t7j7f for pod openshift-network-diagnostics/network-check-target-btw92: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 00:40:31.672063 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:40:31.671989    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1a9f4afb-53aa-401c-9a57-0cfc7e2f6af0-kube-api-access-t7j7f podName:1a9f4afb-53aa-401c-9a57-0cfc7e2f6af0 nodeName:}" failed. No retries permitted until 2026-07-16 00:40:33.671970511 +0000 UTC m=+6.107232230 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-t7j7f" (UniqueName: "kubernetes.io/projected/1a9f4afb-53aa-401c-9a57-0cfc7e2f6af0-kube-api-access-t7j7f") pod "network-check-target-btw92" (UID: "1a9f4afb-53aa-401c-9a57-0cfc7e2f6af0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 00:40:31.955051 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:31.954958    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-15 00:35:29 +0000 UTC" deadline="2028-01-25 14:57:24.267613253 +0000 UTC"
Jul 16 00:40:31.955051 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:31.955001    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13406h16m52.312617221s"
Jul 16 00:40:32.012406 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:32.011376    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-btw92"
Jul 16 00:40:32.012406 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:40:32.011492    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-btw92" podUID="1a9f4afb-53aa-401c-9a57-0cfc7e2f6af0"
Jul 16 00:40:32.012406 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:32.012120    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hgcmx"
Jul 16 00:40:32.012406 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:40:32.012210    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hgcmx" podUID="16a7fa19-0ce5-49ca-96ef-dde5ae694dc0"
Jul 16 00:40:32.012406 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:32.012277    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bkf2h"
Jul 16 00:40:32.012406 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:40:32.012353    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bkf2h" podUID="051e94d1-76fa-484b-b118-b29951f4c5ae"
Jul 16 00:40:32.031538 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:32.031444    2563 generic.go:356] "Generic (PLEG): container finished" podID="69184a0cad720aed3f473960e4918c4c" containerID="7ca5d170cb994b7b4553d63a27d256b20c08de0fe45c43ac83c865b8213fa825" exitCode=0
Jul 16 00:40:32.031538 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:32.031538    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-65.ec2.internal" event={"ID":"69184a0cad720aed3f473960e4918c4c","Type":"ContainerDied","Data":"7ca5d170cb994b7b4553d63a27d256b20c08de0fe45c43ac83c865b8213fa825"}
Jul 16 00:40:32.049542 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:32.049515    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-h4ks8" event={"ID":"090eddb9-3f4c-4a94-9b62-4f6bf8f38428","Type":"ContainerStarted","Data":"fc1ad67418b847df1eecd1a6d9f98b8af023c6e9c17bc8cc0ae685f63f1881ca"}
Jul 16 00:40:32.058671 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:32.058641    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-rk85k" event={"ID":"f0b6d16f-9a13-480c-b981-6481983dbd91","Type":"ContainerStarted","Data":"9fca5c9df05eba226d64326f5ff2fb7fc03872e79c1f119e0d06c22e25978561"}
Jul 16 00:40:32.064297 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:32.064270    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-ksj4j" event={"ID":"ed5df646-0604-408b-bd6e-a502f0bad8e1","Type":"ContainerStarted","Data":"d750c60ee44ccf6714dc10b8d5c4dda15ac69dd813ecc3dbd6e8dbba64355120"}
Jul 16 00:40:32.066252 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:32.066173    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bn5hr" event={"ID":"e990a21d-614b-4d1d-be54-0df6e44487ed","Type":"ContainerStarted","Data":"6e64da4e2977f0bab815abe75af84bbdcf525aeeef83234717b7393174989b35"}
Jul 16 00:40:32.076184 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:32.076159    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-gzbcl" event={"ID":"bd81358e-a649-4ba2-b63e-d9ba85ee93db","Type":"ContainerStarted","Data":"6c548af5b532ce36d13b4ef1112d8670b48c1c1053a336defbfe04cf24ea6c28"}
Jul 16 00:40:32.087191 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:32.087168    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-65.ec2.internal" event={"ID":"ebeafacc0368100d06658fd0ec2ae27a","Type":"ContainerStarted","Data":"7b8ee8bdcf62f1a44a97ec0728e5d04e0c6fcd3e8d98dbd9b0c1f6ac595fc13f"}
Jul 16 00:40:32.096701 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:32.095678    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-tm5f5" event={"ID":"39f3a671-1715-4dd3-9ca0-0b714dde3281","Type":"ContainerStarted","Data":"d68c55a527029ea4888ccc50e1496eb503e67bb29376a7ac1e3bcdff480b46bc"}
Jul 16 00:40:32.101428 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:32.101350    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-l97c7" event={"ID":"49429502-4592-4b95-a41b-7f831eff48da","Type":"ContainerStarted","Data":"9e7ce299e60abfcdb1252e0df40267644a1f98bfebacbb46db2f2e0fc4e159d2"}
Jul 16 00:40:32.108961 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:32.108936    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rjk6r" event={"ID":"24b5fdac-6964-47c8-aa9e-d268d052c2e3","Type":"ContainerStarted","Data":"0efea6001e9f1f1e7ec997884b1bd43a20833ae3052d0327f82377d9177b5be8"}
Jul 16 00:40:32.120070 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:32.120044    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sfpw7" event={"ID":"5a58757b-87fa-4446-b066-015fb48797cf","Type":"ContainerStarted","Data":"1c75f961bbcc617ff29626dcb8119e6cd9772778ba80226bd5abb402c7c7605e"}
Jul 16 00:40:33.145770 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:33.145733    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-65.ec2.internal" event={"ID":"69184a0cad720aed3f473960e4918c4c","Type":"ContainerStarted","Data":"e7c702ef5d821e5c4c01d5460ebff3c9e2cf3ab2e6075c96a1add6b9cc654b6e"}
Jul 16 00:40:33.161838 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:33.161782    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-65.ec2.internal" podStartSLOduration=4.161762543 podStartE2EDuration="4.161762543s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 00:40:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 00:40:32.105539359 +0000 UTC m=+4.540801099" watchObservedRunningTime="2026-07-16 00:40:33.161762543 +0000 UTC m=+5.597024282"
Jul 16 00:40:33.587793 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:33.587696    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/051e94d1-76fa-484b-b118-b29951f4c5ae-metrics-certs\") pod \"network-metrics-daemon-bkf2h\" (UID: \"051e94d1-76fa-484b-b118-b29951f4c5ae\") " pod="openshift-multus/network-metrics-daemon-bkf2h"
Jul 16 00:40:33.587793 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:33.587778    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/16a7fa19-0ce5-49ca-96ef-dde5ae694dc0-original-pull-secret\") pod \"global-pull-secret-syncer-hgcmx\" (UID: \"16a7fa19-0ce5-49ca-96ef-dde5ae694dc0\") " pod="kube-system/global-pull-secret-syncer-hgcmx"
Jul 16 00:40:33.588016 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:40:33.587949    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 16 00:40:33.588016 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:40:33.588012    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16a7fa19-0ce5-49ca-96ef-dde5ae694dc0-original-pull-secret podName:16a7fa19-0ce5-49ca-96ef-dde5ae694dc0 nodeName:}" failed. No retries permitted until 2026-07-16 00:40:37.587993194 +0000 UTC m=+10.023254927 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/16a7fa19-0ce5-49ca-96ef-dde5ae694dc0-original-pull-secret") pod "global-pull-secret-syncer-hgcmx" (UID: "16a7fa19-0ce5-49ca-96ef-dde5ae694dc0") : object "kube-system"/"original-pull-secret" not registered
Jul 16 00:40:33.588297 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:40:33.588210    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 00:40:33.588297 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:40:33.588258    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/051e94d1-76fa-484b-b118-b29951f4c5ae-metrics-certs podName:051e94d1-76fa-484b-b118-b29951f4c5ae nodeName:}" failed. No retries permitted until 2026-07-16 00:40:37.588243686 +0000 UTC m=+10.023505407 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/051e94d1-76fa-484b-b118-b29951f4c5ae-metrics-certs") pod "network-metrics-daemon-bkf2h" (UID: "051e94d1-76fa-484b-b118-b29951f4c5ae") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 00:40:33.688784 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:33.688173    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7j7f\" (UniqueName: \"kubernetes.io/projected/1a9f4afb-53aa-401c-9a57-0cfc7e2f6af0-kube-api-access-t7j7f\") pod \"network-check-target-btw92\" (UID: \"1a9f4afb-53aa-401c-9a57-0cfc7e2f6af0\") " pod="openshift-network-diagnostics/network-check-target-btw92"
Jul 16 00:40:33.688784 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:40:33.688347    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 16 00:40:33.688784 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:40:33.688367    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 16 00:40:33.688784 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:40:33.688379    2563 projected.go:196] Error preparing data for projected volume kube-api-access-t7j7f for pod openshift-network-diagnostics/network-check-target-btw92: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 00:40:33.688784 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:40:33.688436    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1a9f4afb-53aa-401c-9a57-0cfc7e2f6af0-kube-api-access-t7j7f podName:1a9f4afb-53aa-401c-9a57-0cfc7e2f6af0 nodeName:}" failed. No retries permitted until 2026-07-16 00:40:37.688417165 +0000 UTC m=+10.123678898 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-t7j7f" (UniqueName: "kubernetes.io/projected/1a9f4afb-53aa-401c-9a57-0cfc7e2f6af0-kube-api-access-t7j7f") pod "network-check-target-btw92" (UID: "1a9f4afb-53aa-401c-9a57-0cfc7e2f6af0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 00:40:34.002396 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:34.002319    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hgcmx"
Jul 16 00:40:34.002552 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:40:34.002457    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hgcmx" podUID="16a7fa19-0ce5-49ca-96ef-dde5ae694dc0"
Jul 16 00:40:34.002897 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:34.002856    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bkf2h"
Jul 16 00:40:34.002996 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:40:34.002981    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bkf2h" podUID="051e94d1-76fa-484b-b118-b29951f4c5ae"
Jul 16 00:40:34.003056 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:34.003040    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-btw92"
Jul 16 00:40:34.003124 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:40:34.003104    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-btw92" podUID="1a9f4afb-53aa-401c-9a57-0cfc7e2f6af0"
Jul 16 00:40:36.004294 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:36.004252    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hgcmx"
Jul 16 00:40:36.004743 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:40:36.004378    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hgcmx" podUID="16a7fa19-0ce5-49ca-96ef-dde5ae694dc0"
Jul 16 00:40:36.004743 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:36.004736    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bkf2h"
Jul 16 00:40:36.004889 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:40:36.004846    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bkf2h" podUID="051e94d1-76fa-484b-b118-b29951f4c5ae"
Jul 16 00:40:36.004940 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:36.004917    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-btw92"
Jul 16 00:40:36.005008 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:40:36.004989    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-btw92" podUID="1a9f4afb-53aa-401c-9a57-0cfc7e2f6af0"
Jul 16 00:40:37.622797 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:37.622651    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/051e94d1-76fa-484b-b118-b29951f4c5ae-metrics-certs\") pod \"network-metrics-daemon-bkf2h\" (UID: \"051e94d1-76fa-484b-b118-b29951f4c5ae\") " pod="openshift-multus/network-metrics-daemon-bkf2h"
Jul 16 00:40:37.622797 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:37.622724    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/16a7fa19-0ce5-49ca-96ef-dde5ae694dc0-original-pull-secret\") pod \"global-pull-secret-syncer-hgcmx\" (UID: \"16a7fa19-0ce5-49ca-96ef-dde5ae694dc0\") " pod="kube-system/global-pull-secret-syncer-hgcmx"
Jul 16 00:40:37.623363 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:40:37.622897    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 16 00:40:37.623363 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:40:37.622899    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 00:40:37.623363 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:40:37.622962    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16a7fa19-0ce5-49ca-96ef-dde5ae694dc0-original-pull-secret podName:16a7fa19-0ce5-49ca-96ef-dde5ae694dc0 nodeName:}" failed. No retries permitted until 2026-07-16 00:40:45.622941434 +0000 UTC m=+18.058203152 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/16a7fa19-0ce5-49ca-96ef-dde5ae694dc0-original-pull-secret") pod "global-pull-secret-syncer-hgcmx" (UID: "16a7fa19-0ce5-49ca-96ef-dde5ae694dc0") : object "kube-system"/"original-pull-secret" not registered
Jul 16 00:40:37.623363 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:40:37.622981    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/051e94d1-76fa-484b-b118-b29951f4c5ae-metrics-certs podName:051e94d1-76fa-484b-b118-b29951f4c5ae nodeName:}" failed. No retries permitted until 2026-07-16 00:40:45.622971387 +0000 UTC m=+18.058233105 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/051e94d1-76fa-484b-b118-b29951f4c5ae-metrics-certs") pod "network-metrics-daemon-bkf2h" (UID: "051e94d1-76fa-484b-b118-b29951f4c5ae") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 00:40:37.724111 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:37.724080    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7j7f\" (UniqueName: \"kubernetes.io/projected/1a9f4afb-53aa-401c-9a57-0cfc7e2f6af0-kube-api-access-t7j7f\") pod \"network-check-target-btw92\" (UID: \"1a9f4afb-53aa-401c-9a57-0cfc7e2f6af0\") " pod="openshift-network-diagnostics/network-check-target-btw92"
Jul 16 00:40:37.724274 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:40:37.724246    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 16 00:40:37.724274 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:40:37.724271    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 16 00:40:37.724360 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:40:37.724284    2563 projected.go:196] Error preparing data for projected volume kube-api-access-t7j7f for pod openshift-network-diagnostics/network-check-target-btw92: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 00:40:37.724360 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:40:37.724339    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1a9f4afb-53aa-401c-9a57-0cfc7e2f6af0-kube-api-access-t7j7f podName:1a9f4afb-53aa-401c-9a57-0cfc7e2f6af0 nodeName:}" failed. No retries permitted until 2026-07-16 00:40:45.724320674 +0000 UTC m=+18.159582404 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-t7j7f" (UniqueName: "kubernetes.io/projected/1a9f4afb-53aa-401c-9a57-0cfc7e2f6af0-kube-api-access-t7j7f") pod "network-check-target-btw92" (UID: "1a9f4afb-53aa-401c-9a57-0cfc7e2f6af0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 00:40:38.002447 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:38.001874    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hgcmx"
Jul 16 00:40:38.002447 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:38.001894    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bkf2h"
Jul 16 00:40:38.002447 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:40:38.002049    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bkf2h" podUID="051e94d1-76fa-484b-b118-b29951f4c5ae"
Jul 16 00:40:38.002447 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:38.002068    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-btw92"
Jul 16 00:40:38.002447 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:40:38.002147    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hgcmx" podUID="16a7fa19-0ce5-49ca-96ef-dde5ae694dc0"
Jul 16 00:40:38.002447 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:40:38.002233    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-btw92" podUID="1a9f4afb-53aa-401c-9a57-0cfc7e2f6af0"
Jul 16 00:40:40.000702 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:40.000222    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hgcmx"
Jul 16 00:40:40.000702 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:40.000256    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-btw92"
Jul 16 00:40:40.000702 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:40:40.000364    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hgcmx" podUID="16a7fa19-0ce5-49ca-96ef-dde5ae694dc0"
Jul 16 00:40:40.000702 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:40.000222    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bkf2h"
Jul 16 00:40:40.000702 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:40:40.000504    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bkf2h" podUID="051e94d1-76fa-484b-b118-b29951f4c5ae"
Jul 16 00:40:40.000702 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:40:40.000579    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-btw92" podUID="1a9f4afb-53aa-401c-9a57-0cfc7e2f6af0"
Jul 16 00:40:42.000954 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:42.000920    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bkf2h"
Jul 16 00:40:42.000954 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:42.000949    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hgcmx"
Jul 16 00:40:42.001416 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:42.000920    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-btw92"
Jul 16 00:40:42.001416 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:40:42.001061    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bkf2h" podUID="051e94d1-76fa-484b-b118-b29951f4c5ae"
Jul 16 00:40:42.001416 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:40:42.001181    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-btw92" podUID="1a9f4afb-53aa-401c-9a57-0cfc7e2f6af0"
Jul 16 00:40:42.001416 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:40:42.001310    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hgcmx" podUID="16a7fa19-0ce5-49ca-96ef-dde5ae694dc0"
Jul 16 00:40:44.000993 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:44.000912    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hgcmx"
Jul 16 00:40:44.000993 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:44.000946    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-btw92"
Jul 16 00:40:44.001482 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:44.000919    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bkf2h"
Jul 16 00:40:44.001482 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:40:44.001046    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hgcmx" podUID="16a7fa19-0ce5-49ca-96ef-dde5ae694dc0"
Jul 16 00:40:44.001482 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:40:44.001126    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bkf2h" podUID="051e94d1-76fa-484b-b118-b29951f4c5ae"
Jul 16 00:40:44.001482 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:40:44.001245    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-btw92" podUID="1a9f4afb-53aa-401c-9a57-0cfc7e2f6af0"
Jul 16 00:40:45.683730 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:45.683678    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/051e94d1-76fa-484b-b118-b29951f4c5ae-metrics-certs\") pod \"network-metrics-daemon-bkf2h\" (UID: \"051e94d1-76fa-484b-b118-b29951f4c5ae\") " pod="openshift-multus/network-metrics-daemon-bkf2h"
Jul 16 00:40:45.684201 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:45.683749    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/16a7fa19-0ce5-49ca-96ef-dde5ae694dc0-original-pull-secret\") pod \"global-pull-secret-syncer-hgcmx\" (UID: \"16a7fa19-0ce5-49ca-96ef-dde5ae694dc0\") " pod="kube-system/global-pull-secret-syncer-hgcmx"
Jul 16 00:40:45.684201 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:40:45.683851    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 00:40:45.684201 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:40:45.683919    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 16 00:40:45.684201 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:40:45.683936    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/051e94d1-76fa-484b-b118-b29951f4c5ae-metrics-certs podName:051e94d1-76fa-484b-b118-b29951f4c5ae nodeName:}" failed. No retries permitted until 2026-07-16 00:41:01.683913128 +0000 UTC m=+34.119175048 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/051e94d1-76fa-484b-b118-b29951f4c5ae-metrics-certs") pod "network-metrics-daemon-bkf2h" (UID: "051e94d1-76fa-484b-b118-b29951f4c5ae") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 00:40:45.684201 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:40:45.683977    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16a7fa19-0ce5-49ca-96ef-dde5ae694dc0-original-pull-secret podName:16a7fa19-0ce5-49ca-96ef-dde5ae694dc0 nodeName:}" failed. No retries permitted until 2026-07-16 00:41:01.683958306 +0000 UTC m=+34.119220043 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/16a7fa19-0ce5-49ca-96ef-dde5ae694dc0-original-pull-secret") pod "global-pull-secret-syncer-hgcmx" (UID: "16a7fa19-0ce5-49ca-96ef-dde5ae694dc0") : object "kube-system"/"original-pull-secret" not registered
Jul 16 00:40:45.784686 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:45.784654    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7j7f\" (UniqueName: \"kubernetes.io/projected/1a9f4afb-53aa-401c-9a57-0cfc7e2f6af0-kube-api-access-t7j7f\") pod \"network-check-target-btw92\" (UID: \"1a9f4afb-53aa-401c-9a57-0cfc7e2f6af0\") " pod="openshift-network-diagnostics/network-check-target-btw92"
Jul 16 00:40:45.784840 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:40:45.784799    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 16 00:40:45.784840 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:40:45.784815    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 16 00:40:45.784840 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:40:45.784826    2563 projected.go:196] Error preparing data for projected volume kube-api-access-t7j7f for pod openshift-network-diagnostics/network-check-target-btw92: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 00:40:45.784949 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:40:45.784891    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1a9f4afb-53aa-401c-9a57-0cfc7e2f6af0-kube-api-access-t7j7f podName:1a9f4afb-53aa-401c-9a57-0cfc7e2f6af0 nodeName:}" failed. No retries permitted until 2026-07-16 00:41:01.784874181 +0000 UTC m=+34.220135912 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-t7j7f" (UniqueName: "kubernetes.io/projected/1a9f4afb-53aa-401c-9a57-0cfc7e2f6af0-kube-api-access-t7j7f") pod "network-check-target-btw92" (UID: "1a9f4afb-53aa-401c-9a57-0cfc7e2f6af0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 00:40:46.000272 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:46.000194    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hgcmx"
Jul 16 00:40:46.000272 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:46.000223    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-btw92"
Jul 16 00:40:46.000499 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:46.000194    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bkf2h"
Jul 16 00:40:46.000499 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:40:46.000327    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hgcmx" podUID="16a7fa19-0ce5-49ca-96ef-dde5ae694dc0"
Jul 16 00:40:46.000843 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:40:46.000805    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bkf2h" podUID="051e94d1-76fa-484b-b118-b29951f4c5ae"
Jul 16 00:40:46.000978 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:40:46.000960    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-btw92" podUID="1a9f4afb-53aa-401c-9a57-0cfc7e2f6af0"
Jul 16 00:40:48.001844 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:48.001814    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hgcmx"
Jul 16 00:40:48.002322 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:40:48.001947    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hgcmx" podUID="16a7fa19-0ce5-49ca-96ef-dde5ae694dc0"
Jul 16 00:40:48.002387 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:48.002340    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bkf2h"
Jul 16 00:40:48.002486 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:40:48.002451    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bkf2h" podUID="051e94d1-76fa-484b-b118-b29951f4c5ae"
Jul 16 00:40:48.002624 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:48.002538    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-btw92"
Jul 16 00:40:48.002624 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:40:48.002605    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-btw92" podUID="1a9f4afb-53aa-401c-9a57-0cfc7e2f6af0"
Jul 16 00:40:50.000188 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:49.999971    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hgcmx"
Jul 16 00:40:50.000705 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:40:50.000298    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hgcmx" podUID="16a7fa19-0ce5-49ca-96ef-dde5ae694dc0"
Jul 16 00:40:50.000705 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:50.000657    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bkf2h"
Jul 16 00:40:50.000813 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:40:50.000762    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bkf2h" podUID="051e94d1-76fa-484b-b118-b29951f4c5ae"
Jul 16 00:40:50.001030 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:50.001011    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-btw92"
Jul 16 00:40:50.001124 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:40:50.001105    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-btw92" podUID="1a9f4afb-53aa-401c-9a57-0cfc7e2f6af0"
Jul 16 00:40:50.179629 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:50.179429    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-h4ks8" event={"ID":"090eddb9-3f4c-4a94-9b62-4f6bf8f38428","Type":"ContainerStarted","Data":"840875f76435ce596003dfc2256ded6b5bd0fe3991d59613f59f1f32b8951a46"}
Jul 16 00:40:50.180817 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:50.180791    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-rk85k" event={"ID":"f0b6d16f-9a13-480c-b981-6481983dbd91","Type":"ContainerStarted","Data":"dac472d8e1c8a8911cc31f6f96c1fb73e920f00483a8b796fe67eaa1a64467ba"}
Jul 16 00:40:50.182540 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:50.182521    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bn5hr_e990a21d-614b-4d1d-be54-0df6e44487ed/ovn-acl-logging/0.log"
Jul 16 00:40:50.182906 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:50.182886    2563 generic.go:356] "Generic (PLEG): container finished" podID="e990a21d-614b-4d1d-be54-0df6e44487ed" containerID="cb99ad65b27d87d27fe75de61bba7b55c6d7b1901d7e33f10fba3d6d84606ee2" exitCode=1
Jul 16 00:40:50.182992 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:50.182913    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bn5hr" event={"ID":"e990a21d-614b-4d1d-be54-0df6e44487ed","Type":"ContainerStarted","Data":"8b452f40c2e3bf1e096d33e9871c695777bdef84940b0d05f772549159fdf57f"}
Jul 16 00:40:50.182992 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:50.182967    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bn5hr" event={"ID":"e990a21d-614b-4d1d-be54-0df6e44487ed","Type":"ContainerDied","Data":"cb99ad65b27d87d27fe75de61bba7b55c6d7b1901d7e33f10fba3d6d84606ee2"}
Jul 16 00:40:50.183083 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:50.183011    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bn5hr" event={"ID":"e990a21d-614b-4d1d-be54-0df6e44487ed","Type":"ContainerStarted","Data":"c17cd95ddc12d1df13476b7fd6cb4d41b020685181bdca40e1756e0677b41df4"}
Jul 16 00:40:50.184107 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:50.184091    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-gzbcl" event={"ID":"bd81358e-a649-4ba2-b63e-d9ba85ee93db","Type":"ContainerStarted","Data":"b000410ca952370ce593910119bd5cf9ed735fbb6184ea38d6f951d042a439ad"}
Jul 16 00:40:50.185332 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:50.185313    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-tm5f5" event={"ID":"39f3a671-1715-4dd3-9ca0-0b714dde3281","Type":"ContainerStarted","Data":"0d4495e0ddd3e668e957169f9532ebb173247a05fd105903492261f90a32ec93"}
Jul 16 00:40:50.188351 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:50.188316    2563 generic.go:356] "Generic (PLEG): container finished" podID="49429502-4592-4b95-a41b-7f831eff48da" containerID="152269a0b0e2a40cce8e40a7e11f9c3986d907c78b9192f1d4d026311fbc7aed" exitCode=0
Jul 16 00:40:50.188442 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:50.188398    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-l97c7" event={"ID":"49429502-4592-4b95-a41b-7f831eff48da","Type":"ContainerDied","Data":"152269a0b0e2a40cce8e40a7e11f9c3986d907c78b9192f1d4d026311fbc7aed"}
Jul 16 00:40:50.189720 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:50.189704    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rjk6r" event={"ID":"24b5fdac-6964-47c8-aa9e-d268d052c2e3","Type":"ContainerStarted","Data":"b258fa2be057f8722c91ecc918c2b6ba711916ab9616bac01402e81d3c05148a"}
Jul 16 00:40:50.191556 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:50.191539    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sfpw7" event={"ID":"5a58757b-87fa-4446-b066-015fb48797cf","Type":"ContainerStarted","Data":"37c633e011df65fcd4d54e3d0b83439b9254cf2db154acd3dcef6de42adec032"}
Jul 16 00:40:50.199401 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:50.199364    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-65.ec2.internal" podStartSLOduration=21.199349491 podStartE2EDuration="21.199349491s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 00:40:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 00:40:33.161140137 +0000 UTC m=+5.596401878" watchObservedRunningTime="2026-07-16 00:40:50.199349491 +0000 UTC m=+22.634611231"
Jul 16 00:40:50.199537 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:50.199518    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-h4ks8" podStartSLOduration=3.503476644 podStartE2EDuration="22.199509924s" totalImagesPullingTime="18.69603328s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 00:40:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 00:40:50.199168892 +0000 UTC m=+22.634430631" watchObservedRunningTime="2026-07-16 00:40:50.199509924 +0000 UTC m=+22.634771667"
Jul 16 00:40:50.216121 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:50.216082    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-tm5f5" podStartSLOduration=7.492874088 podStartE2EDuration="22.216068389s" totalImagesPullingTime="14.723194301s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 00:40:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 00:40:50.215340696 +0000 UTC m=+22.650602434" watchObservedRunningTime="2026-07-16 00:40:50.216068389 +0000 UTC m=+22.651330168"
Jul 16 00:40:50.254085 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:50.254041    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-rk85k" podStartSLOduration=3.721567552 podStartE2EDuration="22.254028155s" totalImagesPullingTime="18.532460603s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 00:40:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 00:40:50.231687977 +0000 UTC m=+22.666949715" watchObservedRunningTime="2026-07-16 00:40:50.254028155 +0000 UTC m=+22.689289894"
Jul 16 00:40:50.270284 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:50.270243    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-rjk6r" podStartSLOduration=3.621601014 podStartE2EDuration="22.270229075s" totalImagesPullingTime="18.648628061s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 00:40:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 00:40:50.269769914 +0000 UTC m=+22.705031654" watchObservedRunningTime="2026-07-16 00:40:50.270229075 +0000 UTC m=+22.705490814"
Jul 16 00:40:50.287507 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:50.287467    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-gzbcl" podStartSLOduration=3.762448606 podStartE2EDuration="22.287455055s" totalImagesPullingTime="18.525006449s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 00:40:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 00:40:50.287245525 +0000 UTC m=+22.722507265" watchObservedRunningTime="2026-07-16 00:40:50.287455055 +0000 UTC m=+22.722716794"
Jul 16 00:40:51.114242 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:51.114214    2563 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 16 00:40:51.196443 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:51.196413    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sfpw7" event={"ID":"5a58757b-87fa-4446-b066-015fb48797cf","Type":"ContainerStarted","Data":"969197e876a1a9d403aa79283d3147ebe1172fc91a54e23c759a19a2b85fbbab"}
Jul 16 00:40:51.199421 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:51.199402    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bn5hr_e990a21d-614b-4d1d-be54-0df6e44487ed/ovn-acl-logging/0.log"
Jul 16 00:40:51.199936 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:51.199853    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bn5hr" event={"ID":"e990a21d-614b-4d1d-be54-0df6e44487ed","Type":"ContainerStarted","Data":"108e04985aa9b3fcbcc1c1ffdab760ba56503be82da569809782dd61c5292082"}
Jul 16 00:40:51.200031 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:51.199988    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bn5hr" event={"ID":"e990a21d-614b-4d1d-be54-0df6e44487ed","Type":"ContainerStarted","Data":"8dca652704994a95331fb56898b2b9d7b944d95777cf35f38c63dc0d13a8b720"}
Jul 16 00:40:51.200077 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:51.200033    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bn5hr" event={"ID":"e990a21d-614b-4d1d-be54-0df6e44487ed","Type":"ContainerStarted","Data":"a97ef04e1424b2e48b9baab08ae03f0a0b3508273deed223a8dcca8ef8d929b6"}
Jul 16 00:40:51.912633 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:51.912503    2563 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-16T00:40:51.114235117Z","UUID":"9c6adaab-f099-4f17-bb1f-8cf33c6807d0","Handler":null,"Name":"","Endpoint":""}
Jul 16 00:40:51.914344 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:51.914322    2563 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 16 00:40:51.914474 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:51.914365    2563 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 16 00:40:52.004255 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:52.004189    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hgcmx"
Jul 16 00:40:52.004440 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:52.004188    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-btw92"
Jul 16 00:40:52.004440 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:40:52.004297    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hgcmx" podUID="16a7fa19-0ce5-49ca-96ef-dde5ae694dc0"
Jul 16 00:40:52.004440 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:52.004339    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bkf2h"
Jul 16 00:40:52.004639 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:40:52.004441    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-btw92" podUID="1a9f4afb-53aa-401c-9a57-0cfc7e2f6af0"
Jul 16 00:40:52.004639 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:40:52.004556    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bkf2h" podUID="051e94d1-76fa-484b-b118-b29951f4c5ae"
Jul 16 00:40:52.204332 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:52.204249    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-ksj4j" event={"ID":"ed5df646-0604-408b-bd6e-a502f0bad8e1","Type":"ContainerStarted","Data":"e7105c8ae6211399817fe11de49e1a5a51d95e754097fffbf9eaf433ff761fdc"}
Jul 16 00:40:52.219082 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:52.219036    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-ksj4j" podStartSLOduration=5.701946404 podStartE2EDuration="24.219021316s" totalImagesPullingTime="18.517074912s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 00:40:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 00:40:52.2188345 +0000 UTC m=+24.654096238" watchObservedRunningTime="2026-07-16 00:40:52.219021316 +0000 UTC m=+24.654283072"
Jul 16 00:40:53.195567 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:53.195383    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-tm5f5"
Jul 16 00:40:53.196203 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:53.196183    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-tm5f5"
Jul 16 00:40:53.209465 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:53.209438    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sfpw7" event={"ID":"5a58757b-87fa-4446-b066-015fb48797cf","Type":"ContainerStarted","Data":"5748273509f242bc069d8aac7c98d220428dce80cb65c137cddc0faddc695736"}
Jul 16 00:40:53.210010 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:53.209654    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-tm5f5"
Jul 16 00:40:53.210079 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:53.210064    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-tm5f5"
Jul 16 00:40:53.228084 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:53.228010    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sfpw7" podStartSLOduration=4.449193542 podStartE2EDuration="25.227996362s" totalImagesPullingTime="20.77880282s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 00:40:28 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 00:40:53.227658155 +0000 UTC m=+25.662919897" watchObservedRunningTime="2026-07-16 00:40:53.227996362 +0000 UTC m=+25.663258100"
Jul 16 00:40:54.000234 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:54.000203    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-btw92"
Jul 16 00:40:54.000440 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:54.000201    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bkf2h"
Jul 16 00:40:54.000440 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:40:54.000310    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-btw92" podUID="1a9f4afb-53aa-401c-9a57-0cfc7e2f6af0"
Jul 16 00:40:54.000440 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:54.000213    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hgcmx"
Jul 16 00:40:54.000440 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:40:54.000410    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bkf2h" podUID="051e94d1-76fa-484b-b118-b29951f4c5ae"
Jul 16 00:40:54.000664 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:40:54.000500    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hgcmx" podUID="16a7fa19-0ce5-49ca-96ef-dde5ae694dc0"
Jul 16 00:40:55.215074 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:55.215036    2563 generic.go:356] "Generic (PLEG): container finished" podID="49429502-4592-4b95-a41b-7f831eff48da" containerID="0965a438459133e47692fdca859894a79d789906580fe6a09d430bf3b2cee0ae" exitCode=0
Jul 16 00:40:55.215773 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:55.215120    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-l97c7" event={"ID":"49429502-4592-4b95-a41b-7f831eff48da","Type":"ContainerDied","Data":"0965a438459133e47692fdca859894a79d789906580fe6a09d430bf3b2cee0ae"}
Jul 16 00:40:55.218145 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:55.218129    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bn5hr_e990a21d-614b-4d1d-be54-0df6e44487ed/ovn-acl-logging/0.log"
Jul 16 00:40:55.218492 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:55.218457    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bn5hr" event={"ID":"e990a21d-614b-4d1d-be54-0df6e44487ed","Type":"ContainerStarted","Data":"75cc04a2e1f5cc24d853431c8b231393d64c556a0be0e221148d8be984c4f6d7"}
Jul 16 00:40:56.000992 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:56.000956    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-btw92"
Jul 16 00:40:56.001202 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:56.000956    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hgcmx"
Jul 16 00:40:56.001202 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:40:56.001079    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-btw92" podUID="1a9f4afb-53aa-401c-9a57-0cfc7e2f6af0"
Jul 16 00:40:56.001202 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:56.000956    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bkf2h"
Jul 16 00:40:56.001202 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:40:56.001127    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hgcmx" podUID="16a7fa19-0ce5-49ca-96ef-dde5ae694dc0"
Jul 16 00:40:56.001392 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:40:56.001208    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bkf2h" podUID="051e94d1-76fa-484b-b118-b29951f4c5ae"
Jul 16 00:40:57.224793 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:57.224754    2563 generic.go:356] "Generic (PLEG): container finished" podID="49429502-4592-4b95-a41b-7f831eff48da" containerID="260da577aded30329c69bf611a12c4d87e5053890641750fde00ed0296323801" exitCode=0
Jul 16 00:40:57.225343 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:57.224833    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-l97c7" event={"ID":"49429502-4592-4b95-a41b-7f831eff48da","Type":"ContainerDied","Data":"260da577aded30329c69bf611a12c4d87e5053890641750fde00ed0296323801"}
Jul 16 00:40:57.228207 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:57.228190    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bn5hr_e990a21d-614b-4d1d-be54-0df6e44487ed/ovn-acl-logging/0.log"
Jul 16 00:40:57.228556 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:57.228531    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bn5hr" event={"ID":"e990a21d-614b-4d1d-be54-0df6e44487ed","Type":"ContainerStarted","Data":"c2349464681e087a565c0db81655756d2eeb1bb38e56406ca6a62b35f2582ff4"}
Jul 16 00:40:57.228738 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:57.228725    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bn5hr"
Jul 16 00:40:57.228797 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:57.228783    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bn5hr"
Jul 16 00:40:57.228998 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:57.228983    2563 scope.go:117] "RemoveContainer" containerID="cb99ad65b27d87d27fe75de61bba7b55c6d7b1901d7e33f10fba3d6d84606ee2"
Jul 16 00:40:57.245319 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:57.245298    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bn5hr"
Jul 16 00:40:58.001219 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:58.001187    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hgcmx"
Jul 16 00:40:58.001370 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:40:58.001277    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hgcmx" podUID="16a7fa19-0ce5-49ca-96ef-dde5ae694dc0"
Jul 16 00:40:58.001370 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:58.001302    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-btw92"
Jul 16 00:40:58.001458 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:40:58.001381    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-btw92" podUID="1a9f4afb-53aa-401c-9a57-0cfc7e2f6af0"
Jul 16 00:40:58.001458 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:58.001407    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bkf2h"
Jul 16 00:40:58.001514 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:40:58.001481    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bkf2h" podUID="051e94d1-76fa-484b-b118-b29951f4c5ae"
Jul 16 00:40:58.234588 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:58.234557    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bn5hr_e990a21d-614b-4d1d-be54-0df6e44487ed/ovn-acl-logging/0.log"
Jul 16 00:40:58.234974 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:58.234897    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bn5hr" event={"ID":"e990a21d-614b-4d1d-be54-0df6e44487ed","Type":"ContainerStarted","Data":"00b98656e072ca1b85667fbefa414abb4c6fb787ed6ad2afc8aa4d23db9b1663"}
Jul 16 00:40:58.235123 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:58.235107    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bn5hr"
Jul 16 00:40:58.250081 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:58.250058    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bn5hr"
Jul 16 00:40:58.268475 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:58.268398    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-bn5hr" podStartSLOduration=11.466116098 podStartE2EDuration="30.268383887s" totalImagesPullingTime="18.802267789s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 00:40:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 00:40:58.267460125 +0000 UTC m=+30.702721864" watchObservedRunningTime="2026-07-16 00:40:58.268383887 +0000 UTC m=+30.703645627"
Jul 16 00:40:59.239894 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:59.239844    2563 generic.go:356] "Generic (PLEG): container finished" podID="49429502-4592-4b95-a41b-7f831eff48da" containerID="526bd6090310ba913d90a7c5d8a9762a0efe5a207b4816400af145407df5949c" exitCode=0
Jul 16 00:40:59.240328 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:40:59.239913    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-l97c7" event={"ID":"49429502-4592-4b95-a41b-7f831eff48da","Type":"ContainerDied","Data":"526bd6090310ba913d90a7c5d8a9762a0efe5a207b4816400af145407df5949c"}
Jul 16 00:41:00.000731 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:00.000694    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bkf2h"
Jul 16 00:41:00.000923 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:00.000693    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hgcmx"
Jul 16 00:41:00.000923 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:41:00.000873    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bkf2h" podUID="051e94d1-76fa-484b-b118-b29951f4c5ae"
Jul 16 00:41:00.001045 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:41:00.000921    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hgcmx" podUID="16a7fa19-0ce5-49ca-96ef-dde5ae694dc0"
Jul 16 00:41:00.001045 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:00.000695    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-btw92"
Jul 16 00:41:00.001045 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:41:00.001018    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-btw92" podUID="1a9f4afb-53aa-401c-9a57-0cfc7e2f6af0"
Jul 16 00:41:01.703898 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:01.703666    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/16a7fa19-0ce5-49ca-96ef-dde5ae694dc0-original-pull-secret\") pod \"global-pull-secret-syncer-hgcmx\" (UID: \"16a7fa19-0ce5-49ca-96ef-dde5ae694dc0\") " pod="kube-system/global-pull-secret-syncer-hgcmx"
Jul 16 00:41:01.704301 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:01.703943    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/051e94d1-76fa-484b-b118-b29951f4c5ae-metrics-certs\") pod \"network-metrics-daemon-bkf2h\" (UID: \"051e94d1-76fa-484b-b118-b29951f4c5ae\") " pod="openshift-multus/network-metrics-daemon-bkf2h"
Jul 16 00:41:01.704301 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:41:01.703808    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 16 00:41:01.704301 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:41:01.704012    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16a7fa19-0ce5-49ca-96ef-dde5ae694dc0-original-pull-secret podName:16a7fa19-0ce5-49ca-96ef-dde5ae694dc0 nodeName:}" failed. No retries permitted until 2026-07-16 00:41:33.703992938 +0000 UTC m=+66.139254663 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/16a7fa19-0ce5-49ca-96ef-dde5ae694dc0-original-pull-secret") pod "global-pull-secret-syncer-hgcmx" (UID: "16a7fa19-0ce5-49ca-96ef-dde5ae694dc0") : object "kube-system"/"original-pull-secret" not registered
Jul 16 00:41:01.704301 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:41:01.704046    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 00:41:01.704301 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:41:01.704081    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/051e94d1-76fa-484b-b118-b29951f4c5ae-metrics-certs podName:051e94d1-76fa-484b-b118-b29951f4c5ae nodeName:}" failed. No retries permitted until 2026-07-16 00:41:33.70407012 +0000 UTC m=+66.139331836 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/051e94d1-76fa-484b-b118-b29951f4c5ae-metrics-certs") pod "network-metrics-daemon-bkf2h" (UID: "051e94d1-76fa-484b-b118-b29951f4c5ae") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 00:41:01.804645 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:01.804617    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7j7f\" (UniqueName: \"kubernetes.io/projected/1a9f4afb-53aa-401c-9a57-0cfc7e2f6af0-kube-api-access-t7j7f\") pod \"network-check-target-btw92\" (UID: \"1a9f4afb-53aa-401c-9a57-0cfc7e2f6af0\") " pod="openshift-network-diagnostics/network-check-target-btw92"
Jul 16 00:41:01.804831 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:41:01.804741    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 16 00:41:01.804831 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:41:01.804762    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 16 00:41:01.804831 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:41:01.804772    2563 projected.go:196] Error preparing data for projected volume kube-api-access-t7j7f for pod openshift-network-diagnostics/network-check-target-btw92: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 00:41:01.804831 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:41:01.804825    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1a9f4afb-53aa-401c-9a57-0cfc7e2f6af0-kube-api-access-t7j7f podName:1a9f4afb-53aa-401c-9a57-0cfc7e2f6af0 nodeName:}" failed. No retries permitted until 2026-07-16 00:41:33.804812282 +0000 UTC m=+66.240074012 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-t7j7f" (UniqueName: "kubernetes.io/projected/1a9f4afb-53aa-401c-9a57-0cfc7e2f6af0-kube-api-access-t7j7f") pod "network-check-target-btw92" (UID: "1a9f4afb-53aa-401c-9a57-0cfc7e2f6af0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 00:41:02.000946 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:02.000853    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hgcmx"
Jul 16 00:41:02.000946 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:02.000881    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-btw92"
Jul 16 00:41:02.001173 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:02.000853    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bkf2h"
Jul 16 00:41:02.001173 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:41:02.000987    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hgcmx" podUID="16a7fa19-0ce5-49ca-96ef-dde5ae694dc0"
Jul 16 00:41:02.001173 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:41:02.001098    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bkf2h" podUID="051e94d1-76fa-484b-b118-b29951f4c5ae"
Jul 16 00:41:02.001324 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:41:02.001173    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-btw92" podUID="1a9f4afb-53aa-401c-9a57-0cfc7e2f6af0"
Jul 16 00:41:04.000285 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:04.000194    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hgcmx"
Jul 16 00:41:04.000285 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:04.000212    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bkf2h"
Jul 16 00:41:04.000750 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:41:04.000329    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hgcmx" podUID="16a7fa19-0ce5-49ca-96ef-dde5ae694dc0"
Jul 16 00:41:04.000750 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:41:04.000379    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bkf2h" podUID="051e94d1-76fa-484b-b118-b29951f4c5ae"
Jul 16 00:41:04.000750 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:04.000406    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-btw92"
Jul 16 00:41:04.000750 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:41:04.000472    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-btw92" podUID="1a9f4afb-53aa-401c-9a57-0cfc7e2f6af0"
Jul 16 00:41:04.253762 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:04.253680    2563 generic.go:356] "Generic (PLEG): container finished" podID="49429502-4592-4b95-a41b-7f831eff48da" containerID="9152b44b21008b8a664ff2d2ecd17470d8b69299ca12f7fb88d8a14a2225c79a" exitCode=0
Jul 16 00:41:04.253762 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:04.253747    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-l97c7" event={"ID":"49429502-4592-4b95-a41b-7f831eff48da","Type":"ContainerDied","Data":"9152b44b21008b8a664ff2d2ecd17470d8b69299ca12f7fb88d8a14a2225c79a"}
Jul 16 00:41:05.258456 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:05.258420    2563 generic.go:356] "Generic (PLEG): container finished" podID="49429502-4592-4b95-a41b-7f831eff48da" containerID="81ec7ca437980d151fb3bc15e56f746d802331967055dcb025041d2eafdd4f25" exitCode=0
Jul 16 00:41:05.258834 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:05.258478    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-l97c7" event={"ID":"49429502-4592-4b95-a41b-7f831eff48da","Type":"ContainerDied","Data":"81ec7ca437980d151fb3bc15e56f746d802331967055dcb025041d2eafdd4f25"}
Jul 16 00:41:06.000076 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:06.000048    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bkf2h"
Jul 16 00:41:06.000267 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:06.000155    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hgcmx"
Jul 16 00:41:06.000267 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:41:06.000170    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bkf2h" podUID="051e94d1-76fa-484b-b118-b29951f4c5ae"
Jul 16 00:41:06.000267 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:06.000258    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-btw92"
Jul 16 00:41:06.000481 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:41:06.000250    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hgcmx" podUID="16a7fa19-0ce5-49ca-96ef-dde5ae694dc0"
Jul 16 00:41:06.000481 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:41:06.000338    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-btw92" podUID="1a9f4afb-53aa-401c-9a57-0cfc7e2f6af0"
Jul 16 00:41:06.263755 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:06.263664    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-l97c7" event={"ID":"49429502-4592-4b95-a41b-7f831eff48da","Type":"ContainerStarted","Data":"6e47a9064593b6761cb61a7074e169f427395526cdf5a7c489148b094a677ddb"}
Jul 16 00:41:06.328742 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:06.328686    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-l97c7" podStartSLOduration=8.6441632 podStartE2EDuration="38.328666262s" totalImagesPullingTime="29.684503062s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 00:40:28 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 00:41:06.319770756 +0000 UTC m=+38.755032493" watchObservedRunningTime="2026-07-16 00:41:06.328666262 +0000 UTC m=+38.763928002"
Jul 16 00:41:06.329475 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:06.329455    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-hgcmx"]
Jul 16 00:41:06.329634 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:06.329613    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hgcmx"
Jul 16 00:41:06.329746 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:41:06.329725    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hgcmx" podUID="16a7fa19-0ce5-49ca-96ef-dde5ae694dc0"
Jul 16 00:41:06.334032 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:06.334012    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-btw92"]
Jul 16 00:41:06.334129 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:06.334109    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-btw92"
Jul 16 00:41:06.334214 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:41:06.334195    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-btw92" podUID="1a9f4afb-53aa-401c-9a57-0cfc7e2f6af0"
Jul 16 00:41:06.340216 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:06.340193    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-bkf2h"]
Jul 16 00:41:06.340308 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:06.340280    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bkf2h"
Jul 16 00:41:06.340424 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:41:06.340403    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bkf2h" podUID="051e94d1-76fa-484b-b118-b29951f4c5ae"
Jul 16 00:41:08.001543 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:08.001352    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hgcmx"
Jul 16 00:41:08.002032 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:08.001431    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bkf2h"
Jul 16 00:41:08.002032 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:41:08.001622    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hgcmx" podUID="16a7fa19-0ce5-49ca-96ef-dde5ae694dc0"
Jul 16 00:41:08.002032 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:08.001451    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-btw92"
Jul 16 00:41:08.002032 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:41:08.001697    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bkf2h" podUID="051e94d1-76fa-484b-b118-b29951f4c5ae"
Jul 16 00:41:08.002032 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:41:08.001791    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-btw92" podUID="1a9f4afb-53aa-401c-9a57-0cfc7e2f6af0"
Jul 16 00:41:10.000230 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:10.000193    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bkf2h"
Jul 16 00:41:10.000610 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:10.000193    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hgcmx"
Jul 16 00:41:10.000610 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:41:10.000412    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hgcmx" podUID="16a7fa19-0ce5-49ca-96ef-dde5ae694dc0"
Jul 16 00:41:10.000610 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:10.000193    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-btw92"
Jul 16 00:41:10.000610 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:41:10.000304    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bkf2h" podUID="051e94d1-76fa-484b-b118-b29951f4c5ae"
Jul 16 00:41:10.000610 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:41:10.000514    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-btw92" podUID="1a9f4afb-53aa-401c-9a57-0cfc7e2f6af0"
Jul 16 00:41:10.420036 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:10.420011    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-65.ec2.internal" event="NodeReady"
Jul 16 00:41:10.420177 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:10.420167    2563 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 16 00:41:10.474792 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:10.474759    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5fd576b4b9-pwb8z"]
Jul 16 00:41:10.508397 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:10.508321    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5fd576b4b9-pwb8z"]
Jul 16 00:41:10.508397 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:10.508354    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-9tmlj"]
Jul 16 00:41:10.508561 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:10.508442    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5fd576b4b9-pwb8z"
Jul 16 00:41:10.511618 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:10.511592    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-vczkg\""
Jul 16 00:41:10.511748 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:10.511595    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 16 00:41:10.512334 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:10.512321    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 16 00:41:10.513547 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:10.513533    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 16 00:41:10.518329 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:10.518312    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 16 00:41:10.522123 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:10.522103    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-xnkzv"]
Jul 16 00:41:10.522234 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:10.522215    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9tmlj"
Jul 16 00:41:10.524606 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:10.524586    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 16 00:41:10.524696 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:10.524655    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-ll89j\""
Jul 16 00:41:10.526063 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:10.526043    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 16 00:41:10.542296 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:10.542266    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-xnkzv"]
Jul 16 00:41:10.542296 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:10.542298    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-9tmlj"]
Jul 16 00:41:10.542486 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:10.542400    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-xnkzv"
Jul 16 00:41:10.550405 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:10.550385    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 16 00:41:10.550525 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:10.550437    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 16 00:41:10.550525 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:10.550442    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-tf6x9\""
Jul 16 00:41:10.550733 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:10.550720    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 16 00:41:10.574373 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:10.574347    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d75c83b3-ea33-4e0a-b726-6ea607f587a7-metrics-tls\") pod \"dns-default-9tmlj\" (UID: \"d75c83b3-ea33-4e0a-b726-6ea607f587a7\") " pod="openshift-dns/dns-default-9tmlj"
Jul 16 00:41:10.574373 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:10.574378    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98dgp\" (UniqueName: \"kubernetes.io/projected/d75c83b3-ea33-4e0a-b726-6ea607f587a7-kube-api-access-98dgp\") pod \"dns-default-9tmlj\" (UID: \"d75c83b3-ea33-4e0a-b726-6ea607f587a7\") " pod="openshift-dns/dns-default-9tmlj"
Jul 16 00:41:10.574539 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:10.574396    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fb7q\" (UniqueName: \"kubernetes.io/projected/4c6aeea8-fa7c-44db-89df-7bb67070d188-kube-api-access-7fb7q\") pod \"ingress-canary-xnkzv\" (UID: \"4c6aeea8-fa7c-44db-89df-7bb67070d188\") " pod="openshift-ingress-canary/ingress-canary-xnkzv"
Jul 16 00:41:10.574539 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:10.574414    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4562419f-a9c7-499b-b9c3-c1d4ccad25d1-ca-trust-extracted\") pod \"image-registry-5fd576b4b9-pwb8z\" (UID: \"4562419f-a9c7-499b-b9c3-c1d4ccad25d1\") " pod="openshift-image-registry/image-registry-5fd576b4b9-pwb8z"
Jul 16 00:41:10.574539 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:10.574433    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d75c83b3-ea33-4e0a-b726-6ea607f587a7-config-volume\") pod \"dns-default-9tmlj\" (UID: \"d75c83b3-ea33-4e0a-b726-6ea607f587a7\") " pod="openshift-dns/dns-default-9tmlj"
Jul 16 00:41:10.574539 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:10.574472    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d75c83b3-ea33-4e0a-b726-6ea607f587a7-tmp-dir\") pod \"dns-default-9tmlj\" (UID: \"d75c83b3-ea33-4e0a-b726-6ea607f587a7\") " pod="openshift-dns/dns-default-9tmlj"
Jul 16 00:41:10.574539 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:10.574494    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4562419f-a9c7-499b-b9c3-c1d4ccad25d1-installation-pull-secrets\") pod \"image-registry-5fd576b4b9-pwb8z\" (UID: \"4562419f-a9c7-499b-b9c3-c1d4ccad25d1\") " pod="openshift-image-registry/image-registry-5fd576b4b9-pwb8z"
Jul 16 00:41:10.574539 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:10.574514    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlcdj\" (UniqueName: \"kubernetes.io/projected/4562419f-a9c7-499b-b9c3-c1d4ccad25d1-kube-api-access-zlcdj\") pod \"image-registry-5fd576b4b9-pwb8z\" (UID: \"4562419f-a9c7-499b-b9c3-c1d4ccad25d1\") " pod="openshift-image-registry/image-registry-5fd576b4b9-pwb8z"
Jul 16 00:41:10.574717 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:10.574574    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4562419f-a9c7-499b-b9c3-c1d4ccad25d1-registry-tls\") pod \"image-registry-5fd576b4b9-pwb8z\" (UID: \"4562419f-a9c7-499b-b9c3-c1d4ccad25d1\") " pod="openshift-image-registry/image-registry-5fd576b4b9-pwb8z"
Jul 16 00:41:10.574717 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:10.574637    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4562419f-a9c7-499b-b9c3-c1d4ccad25d1-image-registry-private-configuration\") pod \"image-registry-5fd576b4b9-pwb8z\" (UID: \"4562419f-a9c7-499b-b9c3-c1d4ccad25d1\") " pod="openshift-image-registry/image-registry-5fd576b4b9-pwb8z"
Jul 16 00:41:10.574717 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:10.574674    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4562419f-a9c7-499b-b9c3-c1d4ccad25d1-registry-certificates\") pod \"image-registry-5fd576b4b9-pwb8z\" (UID: \"4562419f-a9c7-499b-b9c3-c1d4ccad25d1\") " pod="openshift-image-registry/image-registry-5fd576b4b9-pwb8z"
Jul 16 00:41:10.574717 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:10.574695    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4562419f-a9c7-499b-b9c3-c1d4ccad25d1-bound-sa-token\") pod \"image-registry-5fd576b4b9-pwb8z\" (UID: \"4562419f-a9c7-499b-b9c3-c1d4ccad25d1\") " pod="openshift-image-registry/image-registry-5fd576b4b9-pwb8z"
Jul 16 00:41:10.574717 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:10.574712    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4c6aeea8-fa7c-44db-89df-7bb67070d188-cert\") pod \"ingress-canary-xnkzv\" (UID: \"4c6aeea8-fa7c-44db-89df-7bb67070d188\") " pod="openshift-ingress-canary/ingress-canary-xnkzv"
Jul 16 00:41:10.574857 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:10.574727    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4562419f-a9c7-499b-b9c3-c1d4ccad25d1-trusted-ca\") pod \"image-registry-5fd576b4b9-pwb8z\" (UID: \"4562419f-a9c7-499b-b9c3-c1d4ccad25d1\") " pod="openshift-image-registry/image-registry-5fd576b4b9-pwb8z"
Jul 16 00:41:10.646062 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:10.646022    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-57rzd"]
Jul 16 00:41:10.662286 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:10.662261    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-57rzd"
Jul 16 00:41:10.668678 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:10.668658    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-6hgb5\""
Jul 16 00:41:10.669015 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:10.668997    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 16 00:41:10.669123 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:10.668998    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 16 00:41:10.669835 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:10.669817    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-57rzd"]
Jul 16 00:41:10.670061 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:10.670042    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 16 00:41:10.670166 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:10.670138    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 16 00:41:10.675108 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:10.675088    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4562419f-a9c7-499b-b9c3-c1d4ccad25d1-registry-tls\") pod \"image-registry-5fd576b4b9-pwb8z\" (UID: \"4562419f-a9c7-499b-b9c3-c1d4ccad25d1\") " pod="openshift-image-registry/image-registry-5fd576b4b9-pwb8z"
Jul 16 00:41:10.675189 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:10.675137    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/908a59b2-e065-4d36-a3a2-8de114feaf35-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-57rzd\" (UID: \"908a59b2-e065-4d36-a3a2-8de114feaf35\") " pod="openshift-insights/insights-runtime-extractor-57rzd"
Jul 16 00:41:10.675189 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:10.675161    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7h7f\" (UniqueName: \"kubernetes.io/projected/908a59b2-e065-4d36-a3a2-8de114feaf35-kube-api-access-b7h7f\") pod \"insights-runtime-extractor-57rzd\" (UID: \"908a59b2-e065-4d36-a3a2-8de114feaf35\") " pod="openshift-insights/insights-runtime-extractor-57rzd"
Jul 16 00:41:10.675251 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:10.675188    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/908a59b2-e065-4d36-a3a2-8de114feaf35-data-volume\") pod \"insights-runtime-extractor-57rzd\" (UID: \"908a59b2-e065-4d36-a3a2-8de114feaf35\") " pod="openshift-insights/insights-runtime-extractor-57rzd"
Jul 16 00:41:10.675251 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:10.675214    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4562419f-a9c7-499b-b9c3-c1d4ccad25d1-image-registry-private-configuration\") pod \"image-registry-5fd576b4b9-pwb8z\" (UID: \"4562419f-a9c7-499b-b9c3-c1d4ccad25d1\") " pod="openshift-image-registry/image-registry-5fd576b4b9-pwb8z"
Jul 16 00:41:10.675319 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:10.675266    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4562419f-a9c7-499b-b9c3-c1d4ccad25d1-registry-certificates\") pod \"image-registry-5fd576b4b9-pwb8z\" (UID: \"4562419f-a9c7-499b-b9c3-c1d4ccad25d1\") " pod="openshift-image-registry/image-registry-5fd576b4b9-pwb8z"
Jul 16 00:41:10.675319 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:10.675291    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4562419f-a9c7-499b-b9c3-c1d4ccad25d1-bound-sa-token\") pod \"image-registry-5fd576b4b9-pwb8z\" (UID: \"4562419f-a9c7-499b-b9c3-c1d4ccad25d1\") " pod="openshift-image-registry/image-registry-5fd576b4b9-pwb8z"
Jul 16 00:41:10.675319 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:10.675313    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4c6aeea8-fa7c-44db-89df-7bb67070d188-cert\") pod \"ingress-canary-xnkzv\" (UID: \"4c6aeea8-fa7c-44db-89df-7bb67070d188\") " pod="openshift-ingress-canary/ingress-canary-xnkzv"
Jul 16 00:41:10.675412 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:10.675334    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4562419f-a9c7-499b-b9c3-c1d4ccad25d1-trusted-ca\") pod \"image-registry-5fd576b4b9-pwb8z\" (UID: \"4562419f-a9c7-499b-b9c3-c1d4ccad25d1\") " pod="openshift-image-registry/image-registry-5fd576b4b9-pwb8z"
Jul 16 00:41:10.675501 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:10.675479    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d75c83b3-ea33-4e0a-b726-6ea607f587a7-metrics-tls\") pod \"dns-default-9tmlj\" (UID: \"d75c83b3-ea33-4e0a-b726-6ea607f587a7\") " pod="openshift-dns/dns-default-9tmlj"
Jul 16 00:41:10.675557 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:10.675521    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98dgp\" (UniqueName: \"kubernetes.io/projected/d75c83b3-ea33-4e0a-b726-6ea607f587a7-kube-api-access-98dgp\") pod \"dns-default-9tmlj\" (UID: \"d75c83b3-ea33-4e0a-b726-6ea607f587a7\") " pod="openshift-dns/dns-default-9tmlj"
Jul 16 00:41:10.675557 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:10.675550    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fb7q\" (UniqueName: \"kubernetes.io/projected/4c6aeea8-fa7c-44db-89df-7bb67070d188-kube-api-access-7fb7q\") pod \"ingress-canary-xnkzv\" (UID: \"4c6aeea8-fa7c-44db-89df-7bb67070d188\") " pod="openshift-ingress-canary/ingress-canary-xnkzv"
Jul 16 00:41:10.675651 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:10.675582    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4562419f-a9c7-499b-b9c3-c1d4ccad25d1-ca-trust-extracted\") pod \"image-registry-5fd576b4b9-pwb8z\" (UID: \"4562419f-a9c7-499b-b9c3-c1d4ccad25d1\") " pod="openshift-image-registry/image-registry-5fd576b4b9-pwb8z"
Jul 16 00:41:10.675651 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:10.675610    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d75c83b3-ea33-4e0a-b726-6ea607f587a7-config-volume\") pod \"dns-default-9tmlj\" (UID: \"d75c83b3-ea33-4e0a-b726-6ea607f587a7\") " pod="openshift-dns/dns-default-9tmlj"
Jul 16 00:41:10.675651 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:10.675632    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d75c83b3-ea33-4e0a-b726-6ea607f587a7-tmp-dir\") pod \"dns-default-9tmlj\" (UID: \"d75c83b3-ea33-4e0a-b726-6ea607f587a7\") " pod="openshift-dns/dns-default-9tmlj"
Jul 16 00:41:10.675804 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:10.675655    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4562419f-a9c7-499b-b9c3-c1d4ccad25d1-installation-pull-secrets\") pod \"image-registry-5fd576b4b9-pwb8z\" (UID: \"4562419f-a9c7-499b-b9c3-c1d4ccad25d1\") " pod="openshift-image-registry/image-registry-5fd576b4b9-pwb8z"
Jul 16 00:41:10.675804 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:10.675693    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/908a59b2-e065-4d36-a3a2-8de114feaf35-crio-socket\") pod \"insights-runtime-extractor-57rzd\" (UID: \"908a59b2-e065-4d36-a3a2-8de114feaf35\") " pod="openshift-insights/insights-runtime-extractor-57rzd"
Jul 16 00:41:10.675804 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:10.675729    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlcdj\" (UniqueName: \"kubernetes.io/projected/4562419f-a9c7-499b-b9c3-c1d4ccad25d1-kube-api-access-zlcdj\") pod \"image-registry-5fd576b4b9-pwb8z\" (UID: \"4562419f-a9c7-499b-b9c3-c1d4ccad25d1\") " pod="openshift-image-registry/image-registry-5fd576b4b9-pwb8z"
Jul 16 00:41:10.675804 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:10.675753    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/908a59b2-e065-4d36-a3a2-8de114feaf35-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-57rzd\" (UID: \"908a59b2-e065-4d36-a3a2-8de114feaf35\") " pod="openshift-insights/insights-runtime-extractor-57rzd"
Jul 16 00:41:10.676348 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:10.676192    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4562419f-a9c7-499b-b9c3-c1d4ccad25d1-registry-certificates\") pod \"image-registry-5fd576b4b9-pwb8z\" (UID: \"4562419f-a9c7-499b-b9c3-c1d4ccad25d1\") " pod="openshift-image-registry/image-registry-5fd576b4b9-pwb8z"
Jul 16 00:41:10.676348 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:10.676312    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4562419f-a9c7-499b-b9c3-c1d4ccad25d1-ca-trust-extracted\") pod \"image-registry-5fd576b4b9-pwb8z\" (UID: \"4562419f-a9c7-499b-b9c3-c1d4ccad25d1\") " pod="openshift-image-registry/image-registry-5fd576b4b9-pwb8z"
Jul 16 00:41:10.676488 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:10.676399    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4562419f-a9c7-499b-b9c3-c1d4ccad25d1-trusted-ca\") pod \"image-registry-5fd576b4b9-pwb8z\" (UID: \"4562419f-a9c7-499b-b9c3-c1d4ccad25d1\") " pod="openshift-image-registry/image-registry-5fd576b4b9-pwb8z"
Jul 16 00:41:10.676744 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:10.676725    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d75c83b3-ea33-4e0a-b726-6ea607f587a7-tmp-dir\") pod \"dns-default-9tmlj\" (UID: \"d75c83b3-ea33-4e0a-b726-6ea607f587a7\") " pod="openshift-dns/dns-default-9tmlj"
Jul 16 00:41:10.676904 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:10.676887    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d75c83b3-ea33-4e0a-b726-6ea607f587a7-config-volume\") pod \"dns-default-9tmlj\" (UID: \"d75c83b3-ea33-4e0a-b726-6ea607f587a7\") " pod="openshift-dns/dns-default-9tmlj"
Jul 16 00:41:10.679274 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:10.679251    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d75c83b3-ea33-4e0a-b726-6ea607f587a7-metrics-tls\") pod \"dns-default-9tmlj\" (UID: \"d75c83b3-ea33-4e0a-b726-6ea607f587a7\") " pod="openshift-dns/dns-default-9tmlj"
Jul 16 00:41:10.679441 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:10.679423    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4562419f-a9c7-499b-b9c3-c1d4ccad25d1-image-registry-private-configuration\") pod \"image-registry-5fd576b4b9-pwb8z\" (UID: \"4562419f-a9c7-499b-b9c3-c1d4ccad25d1\") " pod="openshift-image-registry/image-registry-5fd576b4b9-pwb8z"
Jul 16 00:41:10.679441 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:10.679431    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4c6aeea8-fa7c-44db-89df-7bb67070d188-cert\") pod \"ingress-canary-xnkzv\" (UID: \"4c6aeea8-fa7c-44db-89df-7bb67070d188\") " pod="openshift-ingress-canary/ingress-canary-xnkzv"
Jul 16 00:41:10.679536 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:10.679422    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4562419f-a9c7-499b-b9c3-c1d4ccad25d1-installation-pull-secrets\") pod \"image-registry-5fd576b4b9-pwb8z\" (UID: \"4562419f-a9c7-499b-b9c3-c1d4ccad25d1\") " pod="openshift-image-registry/image-registry-5fd576b4b9-pwb8z"
Jul 16 00:41:10.679536 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:10.679483    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4562419f-a9c7-499b-b9c3-c1d4ccad25d1-registry-tls\") pod \"image-registry-5fd576b4b9-pwb8z\" (UID: \"4562419f-a9c7-499b-b9c3-c1d4ccad25d1\") " pod="openshift-image-registry/image-registry-5fd576b4b9-pwb8z"
Jul 16 00:41:10.691101 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:10.691082    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fb7q\" (UniqueName: \"kubernetes.io/projected/4c6aeea8-fa7c-44db-89df-7bb67070d188-kube-api-access-7fb7q\") pod \"ingress-canary-xnkzv\" (UID: \"4c6aeea8-fa7c-44db-89df-7bb67070d188\") " pod="openshift-ingress-canary/ingress-canary-xnkzv"
Jul 16 00:41:10.696949 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:10.696929    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4562419f-a9c7-499b-b9c3-c1d4ccad25d1-bound-sa-token\") pod \"image-registry-5fd576b4b9-pwb8z\" (UID: \"4562419f-a9c7-499b-b9c3-c1d4ccad25d1\") " pod="openshift-image-registry/image-registry-5fd576b4b9-pwb8z"
Jul 16 00:41:10.697044 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:10.697024    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-98dgp\" (UniqueName: \"kubernetes.io/projected/d75c83b3-ea33-4e0a-b726-6ea607f587a7-kube-api-access-98dgp\") pod \"dns-default-9tmlj\" (UID: \"d75c83b3-ea33-4e0a-b726-6ea607f587a7\") " pod="openshift-dns/dns-default-9tmlj"
Jul 16 00:41:10.697546 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:10.697526    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlcdj\" (UniqueName: \"kubernetes.io/projected/4562419f-a9c7-499b-b9c3-c1d4ccad25d1-kube-api-access-zlcdj\") pod \"image-registry-5fd576b4b9-pwb8z\" (UID: \"4562419f-a9c7-499b-b9c3-c1d4ccad25d1\") " pod="openshift-image-registry/image-registry-5fd576b4b9-pwb8z"
Jul 16 00:41:10.776471 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:10.776377    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/908a59b2-e065-4d36-a3a2-8de114feaf35-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-57rzd\" (UID: \"908a59b2-e065-4d36-a3a2-8de114feaf35\") " pod="openshift-insights/insights-runtime-extractor-57rzd"
Jul 16 00:41:10.776471 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:10.776437    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/908a59b2-e065-4d36-a3a2-8de114feaf35-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-57rzd\" (UID: \"908a59b2-e065-4d36-a3a2-8de114feaf35\") " pod="openshift-insights/insights-runtime-extractor-57rzd"
Jul 16 00:41:10.776689 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:10.776527    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7h7f\" (UniqueName: \"kubernetes.io/projected/908a59b2-e065-4d36-a3a2-8de114feaf35-kube-api-access-b7h7f\") pod \"insights-runtime-extractor-57rzd\" (UID: \"908a59b2-e065-4d36-a3a2-8de114feaf35\") " pod="openshift-insights/insights-runtime-extractor-57rzd"
Jul 16 00:41:10.776689 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:10.776547    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/908a59b2-e065-4d36-a3a2-8de114feaf35-data-volume\") pod \"insights-runtime-extractor-57rzd\" (UID: \"908a59b2-e065-4d36-a3a2-8de114feaf35\") " pod="openshift-insights/insights-runtime-extractor-57rzd"
Jul 16 00:41:10.776689 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:10.776597    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/908a59b2-e065-4d36-a3a2-8de114feaf35-crio-socket\") pod \"insights-runtime-extractor-57rzd\" (UID: \"908a59b2-e065-4d36-a3a2-8de114feaf35\") " pod="openshift-insights/insights-runtime-extractor-57rzd"
Jul 16 00:41:10.776841 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:10.776728    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/908a59b2-e065-4d36-a3a2-8de114feaf35-crio-socket\") pod \"insights-runtime-extractor-57rzd\" (UID: \"908a59b2-e065-4d36-a3a2-8de114feaf35\") " pod="openshift-insights/insights-runtime-extractor-57rzd"
Jul 16 00:41:10.777002 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:10.776979    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/908a59b2-e065-4d36-a3a2-8de114feaf35-data-volume\") pod \"insights-runtime-extractor-57rzd\" (UID: \"908a59b2-e065-4d36-a3a2-8de114feaf35\") " pod="openshift-insights/insights-runtime-extractor-57rzd"
Jul 16 00:41:10.777086 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:10.777064    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/908a59b2-e065-4d36-a3a2-8de114feaf35-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-57rzd\" (UID: \"908a59b2-e065-4d36-a3a2-8de114feaf35\") " pod="openshift-insights/insights-runtime-extractor-57rzd"
Jul 16 00:41:10.778662 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:10.778642    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/908a59b2-e065-4d36-a3a2-8de114feaf35-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-57rzd\" (UID: \"908a59b2-e065-4d36-a3a2-8de114feaf35\") " pod="openshift-insights/insights-runtime-extractor-57rzd"
Jul 16 00:41:10.791136 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:10.791119    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7h7f\" (UniqueName: \"kubernetes.io/projected/908a59b2-e065-4d36-a3a2-8de114feaf35-kube-api-access-b7h7f\") pod \"insights-runtime-extractor-57rzd\" (UID: \"908a59b2-e065-4d36-a3a2-8de114feaf35\") " pod="openshift-insights/insights-runtime-extractor-57rzd"
Jul 16 00:41:10.818185 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:10.818158    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5fd576b4b9-pwb8z"
Jul 16 00:41:10.831450 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:10.831427    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9tmlj"
Jul 16 00:41:10.850583 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:10.850549    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-xnkzv"
Jul 16 00:41:10.971092 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:10.971027    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-57rzd"
Jul 16 00:41:11.071320 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:41:11.071291    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4562419f_a9c7_499b_b9c3_c1d4ccad25d1.slice/crio-600f818a620239285d4031a0ad4324150a2bf6d69b58c1833cf905529048e004 WatchSource:0}: Error finding container 600f818a620239285d4031a0ad4324150a2bf6d69b58c1833cf905529048e004: Status 404 returned error can't find the container with id 600f818a620239285d4031a0ad4324150a2bf6d69b58c1833cf905529048e004
Jul 16 00:41:11.072447 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:11.072425    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-9tmlj"]
Jul 16 00:41:11.073694 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:11.073534    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5fd576b4b9-pwb8z"]
Jul 16 00:41:11.076473 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:41:11.076412    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd75c83b3_ea33_4e0a_b726_6ea607f587a7.slice/crio-9d449b6878926c5621a7f96a686b3a0d143b345bdcf702f31ed813c52c2a20d3 WatchSource:0}: Error finding container 9d449b6878926c5621a7f96a686b3a0d143b345bdcf702f31ed813c52c2a20d3: Status 404 returned error can't find the container with id 9d449b6878926c5621a7f96a686b3a0d143b345bdcf702f31ed813c52c2a20d3
Jul 16 00:41:11.082812 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:11.082789    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-xnkzv"]
Jul 16 00:41:11.089392 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:41:11.089363    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c6aeea8_fa7c_44db_89df_7bb67070d188.slice/crio-3a172b20148ad01400e75c67ab03049e57f6ca05409c60296b13fa6a715e85e6 WatchSource:0}: Error finding container 3a172b20148ad01400e75c67ab03049e57f6ca05409c60296b13fa6a715e85e6: Status 404 returned error can't find the container with id 3a172b20148ad01400e75c67ab03049e57f6ca05409c60296b13fa6a715e85e6
Jul 16 00:41:11.146951 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:11.146771    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-57rzd"]
Jul 16 00:41:11.149936 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:41:11.149898    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod908a59b2_e065_4d36_a3a2_8de114feaf35.slice/crio-740b19674b8f303407c3cd466d69856c3caabeb8edf0cd568e1ecc161cf4ad97 WatchSource:0}: Error finding container 740b19674b8f303407c3cd466d69856c3caabeb8edf0cd568e1ecc161cf4ad97: Status 404 returned error can't find the container with id 740b19674b8f303407c3cd466d69856c3caabeb8edf0cd568e1ecc161cf4ad97
Jul 16 00:41:11.275464 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:11.275426    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-57rzd" event={"ID":"908a59b2-e065-4d36-a3a2-8de114feaf35","Type":"ContainerStarted","Data":"740b19674b8f303407c3cd466d69856c3caabeb8edf0cd568e1ecc161cf4ad97"}
Jul 16 00:41:11.276532 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:11.276506    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9tmlj" event={"ID":"d75c83b3-ea33-4e0a-b726-6ea607f587a7","Type":"ContainerStarted","Data":"9d449b6878926c5621a7f96a686b3a0d143b345bdcf702f31ed813c52c2a20d3"}
Jul 16 00:41:11.277573 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:11.277552    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-xnkzv" event={"ID":"4c6aeea8-fa7c-44db-89df-7bb67070d188","Type":"ContainerStarted","Data":"3a172b20148ad01400e75c67ab03049e57f6ca05409c60296b13fa6a715e85e6"}
Jul 16 00:41:11.278999 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:11.278978    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5fd576b4b9-pwb8z" event={"ID":"4562419f-a9c7-499b-b9c3-c1d4ccad25d1","Type":"ContainerStarted","Data":"08b0e2e824111e4202d257df8e14806672145d607d0b0b2851014bafb4d4b663"}
Jul 16 00:41:11.279084 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:11.279016    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5fd576b4b9-pwb8z" event={"ID":"4562419f-a9c7-499b-b9c3-c1d4ccad25d1","Type":"ContainerStarted","Data":"600f818a620239285d4031a0ad4324150a2bf6d69b58c1833cf905529048e004"}
Jul 16 00:41:11.279136 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:11.279118    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5fd576b4b9-pwb8z"
Jul 16 00:41:11.308820 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:11.308641    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5fd576b4b9-pwb8z" podStartSLOduration=7.308619803 podStartE2EDuration="7.308619803s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 00:41:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 00:41:11.308371277 +0000 UTC m=+43.743633019" watchObservedRunningTime="2026-07-16 00:41:11.308619803 +0000 UTC m=+43.743881544"
Jul 16 00:41:12.008585 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:12.008559    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bkf2h"
Jul 16 00:41:12.009891 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:12.008571    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-btw92"
Jul 16 00:41:12.009891 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:12.008603    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hgcmx"
Jul 16 00:41:12.012949 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:12.012351    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 16 00:41:12.014645 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:12.013729    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 16 00:41:12.014645 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:12.013771    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 16 00:41:12.014645 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:12.013729    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-hn9ps\""
Jul 16 00:41:12.015543 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:12.015107    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 16 00:41:12.015543 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:12.015341    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-vmvq6\""
Jul 16 00:41:18.301736 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:18.301510    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9tmlj" event={"ID":"d75c83b3-ea33-4e0a-b726-6ea607f587a7","Type":"ContainerStarted","Data":"dd9842051c4e112cf414d8a09860bb27f1be6c8170376b7dc8d452a6312dfa87"}
Jul 16 00:41:18.302509 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:18.301771    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-9tmlj"
Jul 16 00:41:18.302509 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:18.301812    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9tmlj" event={"ID":"d75c83b3-ea33-4e0a-b726-6ea607f587a7","Type":"ContainerStarted","Data":"65ea5034c86300a54a5433e3ba7059121843c0b2edbbbfb0c108a5bb5240b2a1"}
Jul 16 00:41:18.303075 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:18.303054    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-xnkzv" event={"ID":"4c6aeea8-fa7c-44db-89df-7bb67070d188","Type":"ContainerStarted","Data":"5845677ca0514f2a6874df9c57528beb71e196f9c19f724c4ca79fcc1248d18a"}
Jul 16 00:41:18.304438 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:18.304418    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-57rzd" event={"ID":"908a59b2-e065-4d36-a3a2-8de114feaf35","Type":"ContainerStarted","Data":"261cd2393a2056705080f69c961dc9abb91dba0f7627332edc8eb1ea777ec2a0"}
Jul 16 00:41:18.318465 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:18.318422    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-9tmlj" podStartSLOduration=1.5554139710000001 podStartE2EDuration="8.318406421s" totalImagesPullingTime="6.76299245s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 00:41:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 00:41:18.317599878 +0000 UTC m=+50.752861616" watchObservedRunningTime="2026-07-16 00:41:18.318406421 +0000 UTC m=+50.753668203"
Jul 16 00:41:18.333855 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:18.333806    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-xnkzv" podStartSLOduration=1.584336586 podStartE2EDuration="8.333791122s" totalImagesPullingTime="6.749454536s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 00:41:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 00:41:18.333006645 +0000 UTC m=+50.768268383" watchObservedRunningTime="2026-07-16 00:41:18.333791122 +0000 UTC m=+50.769052855"
Jul 16 00:41:22.320185 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:22.320151    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-57rzd" event={"ID":"908a59b2-e065-4d36-a3a2-8de114feaf35","Type":"ContainerStarted","Data":"78a9f11e40a2f92d8a365cd660cc622294daf79076db197d7295f405d623da0d"}
Jul 16 00:41:26.035341 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:26.035307    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-sxrn4"]
Jul 16 00:41:26.239141 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:26.239107    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-sxrn4"]
Jul 16 00:41:26.239388 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:26.239167    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-gfrth"]
Jul 16 00:41:26.239388 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:26.239317    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-sxrn4"
Jul 16 00:41:26.242330 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:26.242308    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 16 00:41:26.242453 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:26.242331    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 16 00:41:26.242453 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:26.242348    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Jul 16 00:41:26.242453 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:26.242352    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Jul 16 00:41:26.242658 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:26.242647    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 16 00:41:26.242744 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:26.242701    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-xp22l\""
Jul 16 00:41:26.272968 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:26.272942    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-gfrth"
Jul 16 00:41:26.276351 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:26.276325    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-7s2vh\""
Jul 16 00:41:26.276481 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:26.276350    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 16 00:41:26.276481 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:26.276349    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 16 00:41:26.276892 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:26.276879    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 16 00:41:26.305974 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:26.305909    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jx2n\" (UniqueName: \"kubernetes.io/projected/13f8df0b-6a12-4dca-95f4-ca467c9aad65-kube-api-access-6jx2n\") pod \"openshift-state-metrics-84bd68b896-sxrn4\" (UID: \"13f8df0b-6a12-4dca-95f4-ca467c9aad65\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-sxrn4"
Jul 16 00:41:26.305974 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:26.305940    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/13f8df0b-6a12-4dca-95f4-ca467c9aad65-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-sxrn4\" (UID: \"13f8df0b-6a12-4dca-95f4-ca467c9aad65\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-sxrn4"
Jul 16 00:41:26.305974 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:26.305961    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/13f8df0b-6a12-4dca-95f4-ca467c9aad65-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-sxrn4\" (UID: \"13f8df0b-6a12-4dca-95f4-ca467c9aad65\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-sxrn4"
Jul 16 00:41:26.306190 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:26.306099    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/13f8df0b-6a12-4dca-95f4-ca467c9aad65-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-sxrn4\" (UID: \"13f8df0b-6a12-4dca-95f4-ca467c9aad65\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-sxrn4"
Jul 16 00:41:26.406808 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:26.406774    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vq48v\" (UniqueName: \"kubernetes.io/projected/634869d9-271a-4673-b367-e673b5e7b867-kube-api-access-vq48v\") pod \"node-exporter-gfrth\" (UID: \"634869d9-271a-4673-b367-e673b5e7b867\") " pod="openshift-monitoring/node-exporter-gfrth"
Jul 16 00:41:26.406983 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:26.406859    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/634869d9-271a-4673-b367-e673b5e7b867-metrics-client-ca\") pod \"node-exporter-gfrth\" (UID: \"634869d9-271a-4673-b367-e673b5e7b867\") " pod="openshift-monitoring/node-exporter-gfrth"
Jul 16 00:41:26.406983 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:26.406893    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/634869d9-271a-4673-b367-e673b5e7b867-node-exporter-textfile\") pod \"node-exporter-gfrth\" (UID: \"634869d9-271a-4673-b367-e673b5e7b867\") " pod="openshift-monitoring/node-exporter-gfrth"
Jul 16 00:41:26.406983 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:26.406914    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/634869d9-271a-4673-b367-e673b5e7b867-sys\") pod \"node-exporter-gfrth\" (UID: \"634869d9-271a-4673-b367-e673b5e7b867\") " pod="openshift-monitoring/node-exporter-gfrth"
Jul 16 00:41:26.406983 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:26.406954    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/634869d9-271a-4673-b367-e673b5e7b867-node-exporter-tls\") pod \"node-exporter-gfrth\" (UID: \"634869d9-271a-4673-b367-e673b5e7b867\") " pod="openshift-monitoring/node-exporter-gfrth"
Jul 16 00:41:26.407128 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:26.407001    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/634869d9-271a-4673-b367-e673b5e7b867-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gfrth\" (UID: \"634869d9-271a-4673-b367-e673b5e7b867\") " pod="openshift-monitoring/node-exporter-gfrth"
Jul 16 00:41:26.407128 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:26.407051    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/13f8df0b-6a12-4dca-95f4-ca467c9aad65-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-sxrn4\" (UID: \"13f8df0b-6a12-4dca-95f4-ca467c9aad65\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-sxrn4"
Jul 16 00:41:26.407128 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:26.407074    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/634869d9-271a-4673-b367-e673b5e7b867-node-exporter-wtmp\") pod \"node-exporter-gfrth\" (UID: \"634869d9-271a-4673-b367-e673b5e7b867\") " pod="openshift-monitoring/node-exporter-gfrth"
Jul 16 00:41:26.407128 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:26.407092    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/634869d9-271a-4673-b367-e673b5e7b867-node-exporter-accelerators-collector-config\") pod \"node-exporter-gfrth\" (UID: \"634869d9-271a-4673-b367-e673b5e7b867\") " pod="openshift-monitoring/node-exporter-gfrth"
Jul 16 00:41:26.407128 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:26.407111    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/13f8df0b-6a12-4dca-95f4-ca467c9aad65-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-sxrn4\" (UID: \"13f8df0b-6a12-4dca-95f4-ca467c9aad65\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-sxrn4"
Jul 16 00:41:26.407348 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:26.407139    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jx2n\" (UniqueName: \"kubernetes.io/projected/13f8df0b-6a12-4dca-95f4-ca467c9aad65-kube-api-access-6jx2n\") pod \"openshift-state-metrics-84bd68b896-sxrn4\" (UID: \"13f8df0b-6a12-4dca-95f4-ca467c9aad65\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-sxrn4"
Jul 16 00:41:26.407348 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:26.407286    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/13f8df0b-6a12-4dca-95f4-ca467c9aad65-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-sxrn4\" (UID: \"13f8df0b-6a12-4dca-95f4-ca467c9aad65\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-sxrn4"
Jul 16 00:41:26.407348 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:26.407307    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/634869d9-271a-4673-b367-e673b5e7b867-root\") pod \"node-exporter-gfrth\" (UID: \"634869d9-271a-4673-b367-e673b5e7b867\") " pod="openshift-monitoring/node-exporter-gfrth"
Jul 16 00:41:26.407821 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:26.407801    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/13f8df0b-6a12-4dca-95f4-ca467c9aad65-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-sxrn4\" (UID: \"13f8df0b-6a12-4dca-95f4-ca467c9aad65\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-sxrn4"
Jul 16 00:41:26.410963 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:26.410940    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/13f8df0b-6a12-4dca-95f4-ca467c9aad65-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-sxrn4\" (UID: \"13f8df0b-6a12-4dca-95f4-ca467c9aad65\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-sxrn4"
Jul 16 00:41:26.411074 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:26.410973    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/13f8df0b-6a12-4dca-95f4-ca467c9aad65-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-sxrn4\" (UID: \"13f8df0b-6a12-4dca-95f4-ca467c9aad65\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-sxrn4"
Jul 16 00:41:26.417227 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:26.417209    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jx2n\" (UniqueName: \"kubernetes.io/projected/13f8df0b-6a12-4dca-95f4-ca467c9aad65-kube-api-access-6jx2n\") pod \"openshift-state-metrics-84bd68b896-sxrn4\" (UID: \"13f8df0b-6a12-4dca-95f4-ca467c9aad65\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-sxrn4"
Jul 16 00:41:26.508057 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:26.508024    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/634869d9-271a-4673-b367-e673b5e7b867-metrics-client-ca\") pod \"node-exporter-gfrth\" (UID: \"634869d9-271a-4673-b367-e673b5e7b867\") " pod="openshift-monitoring/node-exporter-gfrth"
Jul 16 00:41:26.508057 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:26.508058    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/634869d9-271a-4673-b367-e673b5e7b867-node-exporter-textfile\") pod \"node-exporter-gfrth\" (UID: \"634869d9-271a-4673-b367-e673b5e7b867\") " pod="openshift-monitoring/node-exporter-gfrth"
Jul 16 00:41:26.508286 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:26.508079    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/634869d9-271a-4673-b367-e673b5e7b867-sys\") pod \"node-exporter-gfrth\" (UID: \"634869d9-271a-4673-b367-e673b5e7b867\") " pod="openshift-monitoring/node-exporter-gfrth"
Jul 16 00:41:26.508286 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:26.508099    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/634869d9-271a-4673-b367-e673b5e7b867-node-exporter-tls\") pod \"node-exporter-gfrth\" (UID: \"634869d9-271a-4673-b367-e673b5e7b867\") " pod="openshift-monitoring/node-exporter-gfrth"
Jul 16 00:41:26.508286 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:26.508121    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/634869d9-271a-4673-b367-e673b5e7b867-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gfrth\" (UID: \"634869d9-271a-4673-b367-e673b5e7b867\") " pod="openshift-monitoring/node-exporter-gfrth"
Jul 16 00:41:26.508286 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:26.508165    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/634869d9-271a-4673-b367-e673b5e7b867-sys\") pod \"node-exporter-gfrth\" (UID: \"634869d9-271a-4673-b367-e673b5e7b867\") " pod="openshift-monitoring/node-exporter-gfrth"
Jul 16 00:41:26.508477 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:26.508284    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/634869d9-271a-4673-b367-e673b5e7b867-node-exporter-wtmp\") pod \"node-exporter-gfrth\" (UID: \"634869d9-271a-4673-b367-e673b5e7b867\") " pod="openshift-monitoring/node-exporter-gfrth"
Jul 16 00:41:26.508477 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:26.508318    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/634869d9-271a-4673-b367-e673b5e7b867-node-exporter-accelerators-collector-config\") pod \"node-exporter-gfrth\" (UID: \"634869d9-271a-4673-b367-e673b5e7b867\") " pod="openshift-monitoring/node-exporter-gfrth"
Jul 16 00:41:26.508477 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:26.508385    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/634869d9-271a-4673-b367-e673b5e7b867-node-exporter-textfile\") pod \"node-exporter-gfrth\" (UID: \"634869d9-271a-4673-b367-e673b5e7b867\") " pod="openshift-monitoring/node-exporter-gfrth"
Jul 16 00:41:26.508477 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:26.508389    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/634869d9-271a-4673-b367-e673b5e7b867-root\") pod \"node-exporter-gfrth\" (UID: \"634869d9-271a-4673-b367-e673b5e7b867\") " pod="openshift-monitoring/node-exporter-gfrth"
Jul 16 00:41:26.508477 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:26.508427    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/634869d9-271a-4673-b367-e673b5e7b867-root\") pod \"node-exporter-gfrth\" (UID: \"634869d9-271a-4673-b367-e673b5e7b867\") " pod="openshift-monitoring/node-exporter-gfrth"
Jul 16 00:41:26.508477 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:26.508447    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vq48v\" (UniqueName: \"kubernetes.io/projected/634869d9-271a-4673-b367-e673b5e7b867-kube-api-access-vq48v\") pod \"node-exporter-gfrth\" (UID: \"634869d9-271a-4673-b367-e673b5e7b867\") " pod="openshift-monitoring/node-exporter-gfrth"
Jul 16 00:41:26.508477 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:26.508450    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/634869d9-271a-4673-b367-e673b5e7b867-node-exporter-wtmp\") pod \"node-exporter-gfrth\" (UID: \"634869d9-271a-4673-b367-e673b5e7b867\") " pod="openshift-monitoring/node-exporter-gfrth"
Jul 16 00:41:26.508832 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:26.508636    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/634869d9-271a-4673-b367-e673b5e7b867-metrics-client-ca\") pod \"node-exporter-gfrth\" (UID: \"634869d9-271a-4673-b367-e673b5e7b867\") " pod="openshift-monitoring/node-exporter-gfrth"
Jul 16 00:41:26.508906 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:26.508833    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/634869d9-271a-4673-b367-e673b5e7b867-node-exporter-accelerators-collector-config\") pod \"node-exporter-gfrth\" (UID: \"634869d9-271a-4673-b367-e673b5e7b867\") " pod="openshift-monitoring/node-exporter-gfrth"
Jul 16 00:41:26.510426 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:26.510396    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/634869d9-271a-4673-b367-e673b5e7b867-node-exporter-tls\") pod \"node-exporter-gfrth\" (UID: \"634869d9-271a-4673-b367-e673b5e7b867\") " pod="openshift-monitoring/node-exporter-gfrth"
Jul 16 00:41:26.510426 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:26.510417    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/634869d9-271a-4673-b367-e673b5e7b867-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gfrth\" (UID: \"634869d9-271a-4673-b367-e673b5e7b867\") " pod="openshift-monitoring/node-exporter-gfrth"
Jul 16 00:41:26.519835 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:26.519818    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vq48v\" (UniqueName: \"kubernetes.io/projected/634869d9-271a-4673-b367-e673b5e7b867-kube-api-access-vq48v\") pod \"node-exporter-gfrth\" (UID: \"634869d9-271a-4673-b367-e673b5e7b867\") " pod="openshift-monitoring/node-exporter-gfrth"
Jul 16 00:41:26.547787 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:26.547768    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-sxrn4"
Jul 16 00:41:26.581542 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:26.581523    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-gfrth"
Jul 16 00:41:26.711076 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:41:26.710975    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod634869d9_271a_4673_b367_e673b5e7b867.slice/crio-bf94f712c770717b0380b4eb6030cf6fac38ef600638f48464b6d3acaf9f5c7a WatchSource:0}: Error finding container bf94f712c770717b0380b4eb6030cf6fac38ef600638f48464b6d3acaf9f5c7a: Status 404 returned error can't find the container with id bf94f712c770717b0380b4eb6030cf6fac38ef600638f48464b6d3acaf9f5c7a
Jul 16 00:41:26.834434 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:26.834400    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-sxrn4"]
Jul 16 00:41:26.837959 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:41:26.837931    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13f8df0b_6a12_4dca_95f4_ca467c9aad65.slice/crio-32a6dbc25216cd879528f3efdb9a882c41b753195e61ba540b163c999b46d67e WatchSource:0}: Error finding container 32a6dbc25216cd879528f3efdb9a882c41b753195e61ba540b163c999b46d67e: Status 404 returned error can't find the container with id 32a6dbc25216cd879528f3efdb9a882c41b753195e61ba540b163c999b46d67e
Jul 16 00:41:27.165735 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:27.165663    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 16 00:41:27.168205 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:27.168190    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 16 00:41:27.170922 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:27.170850    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 16 00:41:27.171041 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:27.170923    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 16 00:41:27.171041 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:27.170941    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 16 00:41:27.171041 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:27.170929    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 16 00:41:27.171041 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:27.170925    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 16 00:41:27.171275 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:27.171262    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 16 00:41:27.171327 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:27.171270    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 16 00:41:27.171378 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:27.171332    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 16 00:41:27.171378 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:27.171367    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 16 00:41:27.171508 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:27.171494    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-lqhwj\""
Jul 16 00:41:27.190786 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:27.190759    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 16 00:41:27.213916 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:27.213891    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6bcbf625-91d3-456a-aa3d-7875407ec889-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6bcbf625-91d3-456a-aa3d-7875407ec889\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 00:41:27.214021 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:27.213922    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6bcbf625-91d3-456a-aa3d-7875407ec889-web-config\") pod \"alertmanager-main-0\" (UID: \"6bcbf625-91d3-456a-aa3d-7875407ec889\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 00:41:27.214021 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:27.213939    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6bcbf625-91d3-456a-aa3d-7875407ec889-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6bcbf625-91d3-456a-aa3d-7875407ec889\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 00:41:27.214021 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:27.214001    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6bcbf625-91d3-456a-aa3d-7875407ec889-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6bcbf625-91d3-456a-aa3d-7875407ec889\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 00:41:27.214121 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:27.214044    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6bcbf625-91d3-456a-aa3d-7875407ec889-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6bcbf625-91d3-456a-aa3d-7875407ec889\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 00:41:27.214121 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:27.214097    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6bcbf625-91d3-456a-aa3d-7875407ec889-config-volume\") pod \"alertmanager-main-0\" (UID: \"6bcbf625-91d3-456a-aa3d-7875407ec889\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 00:41:27.214182 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:27.214118    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6bcbf625-91d3-456a-aa3d-7875407ec889-config-out\") pod \"alertmanager-main-0\" (UID: \"6bcbf625-91d3-456a-aa3d-7875407ec889\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 00:41:27.214182 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:27.214144    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6bcbf625-91d3-456a-aa3d-7875407ec889-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6bcbf625-91d3-456a-aa3d-7875407ec889\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 00:41:27.214182 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:27.214161    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6bcbf625-91d3-456a-aa3d-7875407ec889-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6bcbf625-91d3-456a-aa3d-7875407ec889\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 00:41:27.214271 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:27.214225    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6bcbf625-91d3-456a-aa3d-7875407ec889-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6bcbf625-91d3-456a-aa3d-7875407ec889\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 00:41:27.214271 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:27.214261    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6bcbf625-91d3-456a-aa3d-7875407ec889-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6bcbf625-91d3-456a-aa3d-7875407ec889\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 00:41:27.214328 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:27.214283    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dq97l\" (UniqueName: \"kubernetes.io/projected/6bcbf625-91d3-456a-aa3d-7875407ec889-kube-api-access-dq97l\") pod \"alertmanager-main-0\" (UID: \"6bcbf625-91d3-456a-aa3d-7875407ec889\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 00:41:27.214328 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:27.214309    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6bcbf625-91d3-456a-aa3d-7875407ec889-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6bcbf625-91d3-456a-aa3d-7875407ec889\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 00:41:27.315498 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:27.315465    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6bcbf625-91d3-456a-aa3d-7875407ec889-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6bcbf625-91d3-456a-aa3d-7875407ec889\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 00:41:27.315678 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:27.315504    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6bcbf625-91d3-456a-aa3d-7875407ec889-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6bcbf625-91d3-456a-aa3d-7875407ec889\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 00:41:27.315678 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:27.315569    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6bcbf625-91d3-456a-aa3d-7875407ec889-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6bcbf625-91d3-456a-aa3d-7875407ec889\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 00:41:27.315678 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:27.315605    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6bcbf625-91d3-456a-aa3d-7875407ec889-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6bcbf625-91d3-456a-aa3d-7875407ec889\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 00:41:27.315678 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:27.315633    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dq97l\" (UniqueName: \"kubernetes.io/projected/6bcbf625-91d3-456a-aa3d-7875407ec889-kube-api-access-dq97l\") pod \"alertmanager-main-0\" (UID: \"6bcbf625-91d3-456a-aa3d-7875407ec889\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 00:41:27.315678 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:41:27.315643    2563 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found
Jul 16 00:41:27.315678 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:27.315677    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6bcbf625-91d3-456a-aa3d-7875407ec889-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6bcbf625-91d3-456a-aa3d-7875407ec889\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 00:41:27.315999 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:27.315710    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6bcbf625-91d3-456a-aa3d-7875407ec889-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6bcbf625-91d3-456a-aa3d-7875407ec889\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 00:41:27.315999 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:41:27.315737    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6bcbf625-91d3-456a-aa3d-7875407ec889-secret-alertmanager-main-tls podName:6bcbf625-91d3-456a-aa3d-7875407ec889 nodeName:}" failed. No retries permitted until 2026-07-16 00:41:27.815715176 +0000 UTC m=+60.250976901 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/6bcbf625-91d3-456a-aa3d-7875407ec889-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "6bcbf625-91d3-456a-aa3d-7875407ec889") : secret "alertmanager-main-tls" not found
Jul 16 00:41:27.315999 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:41:27.315763    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6bcbf625-91d3-456a-aa3d-7875407ec889-alertmanager-trusted-ca-bundle podName:6bcbf625-91d3-456a-aa3d-7875407ec889 nodeName:}" failed. No retries permitted until 2026-07-16 00:41:27.815753292 +0000 UTC m=+60.251015013 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/6bcbf625-91d3-456a-aa3d-7875407ec889-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "6bcbf625-91d3-456a-aa3d-7875407ec889") : configmap references non-existent config key: ca-bundle.crt
Jul 16 00:41:27.315999 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:27.315803    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6bcbf625-91d3-456a-aa3d-7875407ec889-web-config\") pod \"alertmanager-main-0\" (UID: \"6bcbf625-91d3-456a-aa3d-7875407ec889\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 00:41:27.315999 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:27.315835    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6bcbf625-91d3-456a-aa3d-7875407ec889-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6bcbf625-91d3-456a-aa3d-7875407ec889\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 00:41:27.315999 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:27.315894    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6bcbf625-91d3-456a-aa3d-7875407ec889-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6bcbf625-91d3-456a-aa3d-7875407ec889\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 00:41:27.315999 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:27.315927    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6bcbf625-91d3-456a-aa3d-7875407ec889-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6bcbf625-91d3-456a-aa3d-7875407ec889\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 00:41:27.315999 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:27.315978    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6bcbf625-91d3-456a-aa3d-7875407ec889-config-volume\") pod \"alertmanager-main-0\" (UID: \"6bcbf625-91d3-456a-aa3d-7875407ec889\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 00:41:27.315999 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:27.316001    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6bcbf625-91d3-456a-aa3d-7875407ec889-config-out\") pod \"alertmanager-main-0\" (UID: \"6bcbf625-91d3-456a-aa3d-7875407ec889\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 00:41:27.316488 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:27.316464    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6bcbf625-91d3-456a-aa3d-7875407ec889-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6bcbf625-91d3-456a-aa3d-7875407ec889\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 00:41:27.316591 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:27.316567    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6bcbf625-91d3-456a-aa3d-7875407ec889-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6bcbf625-91d3-456a-aa3d-7875407ec889\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 00:41:27.318586 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:27.318541    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6bcbf625-91d3-456a-aa3d-7875407ec889-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6bcbf625-91d3-456a-aa3d-7875407ec889\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 00:41:27.318686 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:27.318611    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6bcbf625-91d3-456a-aa3d-7875407ec889-web-config\") pod \"alertmanager-main-0\" (UID: \"6bcbf625-91d3-456a-aa3d-7875407ec889\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 00:41:27.318765 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:27.318745    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6bcbf625-91d3-456a-aa3d-7875407ec889-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6bcbf625-91d3-456a-aa3d-7875407ec889\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 00:41:27.318990 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:27.318969    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6bcbf625-91d3-456a-aa3d-7875407ec889-config-out\") pod \"alertmanager-main-0\" (UID: \"6bcbf625-91d3-456a-aa3d-7875407ec889\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 00:41:27.319428 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:27.319398    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6bcbf625-91d3-456a-aa3d-7875407ec889-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6bcbf625-91d3-456a-aa3d-7875407ec889\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 00:41:27.320383 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:27.320360    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6bcbf625-91d3-456a-aa3d-7875407ec889-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6bcbf625-91d3-456a-aa3d-7875407ec889\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 00:41:27.320816 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:27.320796    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6bcbf625-91d3-456a-aa3d-7875407ec889-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6bcbf625-91d3-456a-aa3d-7875407ec889\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 00:41:27.321411 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:27.321388    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6bcbf625-91d3-456a-aa3d-7875407ec889-config-volume\") pod \"alertmanager-main-0\" (UID: \"6bcbf625-91d3-456a-aa3d-7875407ec889\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 00:41:27.326242 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:27.326217    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dq97l\" (UniqueName: \"kubernetes.io/projected/6bcbf625-91d3-456a-aa3d-7875407ec889-kube-api-access-dq97l\") pod \"alertmanager-main-0\" (UID: \"6bcbf625-91d3-456a-aa3d-7875407ec889\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 00:41:27.339416 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:27.339382    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gfrth" event={"ID":"634869d9-271a-4673-b367-e673b5e7b867","Type":"ContainerStarted","Data":"bf94f712c770717b0380b4eb6030cf6fac38ef600638f48464b6d3acaf9f5c7a"}
Jul 16 00:41:27.342006 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:27.341964    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-57rzd" event={"ID":"908a59b2-e065-4d36-a3a2-8de114feaf35","Type":"ContainerStarted","Data":"b344225464a6df1f359395b7c68eb81274c4ef05d97ec99237b55bdd68b476af"}
Jul 16 00:41:27.344035 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:27.344010    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-sxrn4" event={"ID":"13f8df0b-6a12-4dca-95f4-ca467c9aad65","Type":"ContainerStarted","Data":"438d62627e56dc65c7a1432e78dcc587520e2870eafddbd77686afdd9517baf1"}
Jul 16 00:41:27.344133 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:27.344065    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-sxrn4" event={"ID":"13f8df0b-6a12-4dca-95f4-ca467c9aad65","Type":"ContainerStarted","Data":"8e9e7ac58d6e28415eceb60a4945937470d3d211458995a5e11a66cde2c343b9"}
Jul 16 00:41:27.344133 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:27.344093    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-sxrn4" event={"ID":"13f8df0b-6a12-4dca-95f4-ca467c9aad65","Type":"ContainerStarted","Data":"32a6dbc25216cd879528f3efdb9a882c41b753195e61ba540b163c999b46d67e"}
Jul 16 00:41:27.365125 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:27.365052    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-57rzd" podStartSLOduration=1.9791002340000001 podStartE2EDuration="17.365037237s" totalImagesPullingTime="15.385937003s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 00:41:10 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 00:41:27.363763066 +0000 UTC m=+59.799024806" watchObservedRunningTime="2026-07-16 00:41:27.365037237 +0000 UTC m=+59.800298976"
Jul 16 00:41:27.821479 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:27.821308    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6bcbf625-91d3-456a-aa3d-7875407ec889-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6bcbf625-91d3-456a-aa3d-7875407ec889\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 00:41:27.821479 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:27.821390    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6bcbf625-91d3-456a-aa3d-7875407ec889-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6bcbf625-91d3-456a-aa3d-7875407ec889\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 00:41:27.821479 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:41:27.821482    2563 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found
Jul 16 00:41:27.821734 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:41:27.821574    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6bcbf625-91d3-456a-aa3d-7875407ec889-secret-alertmanager-main-tls podName:6bcbf625-91d3-456a-aa3d-7875407ec889 nodeName:}" failed. No retries permitted until 2026-07-16 00:41:28.821553039 +0000 UTC m=+61.256814755 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/6bcbf625-91d3-456a-aa3d-7875407ec889-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "6bcbf625-91d3-456a-aa3d-7875407ec889") : secret "alertmanager-main-tls" not found
Jul 16 00:41:27.822314 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:27.822273    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6bcbf625-91d3-456a-aa3d-7875407ec889-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6bcbf625-91d3-456a-aa3d-7875407ec889\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 00:41:28.049174 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:28.049142    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-7f7dd99658-8wp4p"]
Jul 16 00:41:28.052579 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:28.052557    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7f7dd99658-8wp4p"
Jul 16 00:41:28.056544 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:28.056266    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Jul 16 00:41:28.056544 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:28.056314    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Jul 16 00:41:28.056544 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:28.056349    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Jul 16 00:41:28.056941 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:28.056902    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Jul 16 00:41:28.057044 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:28.056982    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-7a50sv14iqro5\""
Jul 16 00:41:28.057081 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:28.056908    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Jul 16 00:41:28.057081 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:28.057064    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-ldrhh\""
Jul 16 00:41:28.071719 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:28.071648    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7f7dd99658-8wp4p"]
Jul 16 00:41:28.125026 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:28.124991    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/15d0414e-398c-40e6-aa30-ede0844470e8-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7f7dd99658-8wp4p\" (UID: \"15d0414e-398c-40e6-aa30-ede0844470e8\") " pod="openshift-monitoring/thanos-querier-7f7dd99658-8wp4p"
Jul 16 00:41:28.125182 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:28.125033    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/15d0414e-398c-40e6-aa30-ede0844470e8-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7f7dd99658-8wp4p\" (UID: \"15d0414e-398c-40e6-aa30-ede0844470e8\") " pod="openshift-monitoring/thanos-querier-7f7dd99658-8wp4p"
Jul 16 00:41:28.125182 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:28.125156    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/15d0414e-398c-40e6-aa30-ede0844470e8-metrics-client-ca\") pod \"thanos-querier-7f7dd99658-8wp4p\" (UID: \"15d0414e-398c-40e6-aa30-ede0844470e8\") " pod="openshift-monitoring/thanos-querier-7f7dd99658-8wp4p"
Jul 16 00:41:28.125269 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:28.125185    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/15d0414e-398c-40e6-aa30-ede0844470e8-secret-grpc-tls\") pod \"thanos-querier-7f7dd99658-8wp4p\" (UID: \"15d0414e-398c-40e6-aa30-ede0844470e8\") " pod="openshift-monitoring/thanos-querier-7f7dd99658-8wp4p"
Jul 16 00:41:28.125269 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:28.125220    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/15d0414e-398c-40e6-aa30-ede0844470e8-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7f7dd99658-8wp4p\" (UID: \"15d0414e-398c-40e6-aa30-ede0844470e8\") " pod="openshift-monitoring/thanos-querier-7f7dd99658-8wp4p"
Jul 16 00:41:28.125348 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:28.125278    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/15d0414e-398c-40e6-aa30-ede0844470e8-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7f7dd99658-8wp4p\" (UID: \"15d0414e-398c-40e6-aa30-ede0844470e8\") " pod="openshift-monitoring/thanos-querier-7f7dd99658-8wp4p"
Jul 16 00:41:28.125348 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:28.125308    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mt7qv\" (UniqueName: \"kubernetes.io/projected/15d0414e-398c-40e6-aa30-ede0844470e8-kube-api-access-mt7qv\") pod \"thanos-querier-7f7dd99658-8wp4p\" (UID: \"15d0414e-398c-40e6-aa30-ede0844470e8\") " pod="openshift-monitoring/thanos-querier-7f7dd99658-8wp4p"
Jul 16 00:41:28.125348 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:28.125327    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/15d0414e-398c-40e6-aa30-ede0844470e8-secret-thanos-querier-tls\") pod \"thanos-querier-7f7dd99658-8wp4p\" (UID: \"15d0414e-398c-40e6-aa30-ede0844470e8\") " pod="openshift-monitoring/thanos-querier-7f7dd99658-8wp4p"
Jul 16 00:41:28.225856 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:28.225831    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/15d0414e-398c-40e6-aa30-ede0844470e8-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7f7dd99658-8wp4p\" (UID: \"15d0414e-398c-40e6-aa30-ede0844470e8\") " pod="openshift-monitoring/thanos-querier-7f7dd99658-8wp4p"
Jul 16 00:41:28.226219 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:28.225890    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/15d0414e-398c-40e6-aa30-ede0844470e8-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7f7dd99658-8wp4p\" (UID: \"15d0414e-398c-40e6-aa30-ede0844470e8\") " pod="openshift-monitoring/thanos-querier-7f7dd99658-8wp4p"
Jul 16 00:41:28.226219 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:28.225911    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mt7qv\" (UniqueName: \"kubernetes.io/projected/15d0414e-398c-40e6-aa30-ede0844470e8-kube-api-access-mt7qv\") pod \"thanos-querier-7f7dd99658-8wp4p\" (UID: \"15d0414e-398c-40e6-aa30-ede0844470e8\") " pod="openshift-monitoring/thanos-querier-7f7dd99658-8wp4p"
Jul 16 00:41:28.226219 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:28.225931    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/15d0414e-398c-40e6-aa30-ede0844470e8-secret-thanos-querier-tls\") pod \"thanos-querier-7f7dd99658-8wp4p\" (UID: \"15d0414e-398c-40e6-aa30-ede0844470e8\") " pod="openshift-monitoring/thanos-querier-7f7dd99658-8wp4p"
Jul 16 00:41:28.226219 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:28.225969    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/15d0414e-398c-40e6-aa30-ede0844470e8-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7f7dd99658-8wp4p\" (UID: \"15d0414e-398c-40e6-aa30-ede0844470e8\") " pod="openshift-monitoring/thanos-querier-7f7dd99658-8wp4p"
Jul 16 00:41:28.226437 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:28.226366    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/15d0414e-398c-40e6-aa30-ede0844470e8-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7f7dd99658-8wp4p\" (UID: \"15d0414e-398c-40e6-aa30-ede0844470e8\") " pod="openshift-monitoring/thanos-querier-7f7dd99658-8wp4p"
Jul 16 00:41:28.227531 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:28.226576    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/15d0414e-398c-40e6-aa30-ede0844470e8-metrics-client-ca\") pod \"thanos-querier-7f7dd99658-8wp4p\" (UID: \"15d0414e-398c-40e6-aa30-ede0844470e8\") " pod="openshift-monitoring/thanos-querier-7f7dd99658-8wp4p"
Jul 16 00:41:28.227531 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:28.226614    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/15d0414e-398c-40e6-aa30-ede0844470e8-secret-grpc-tls\") pod \"thanos-querier-7f7dd99658-8wp4p\" (UID: \"15d0414e-398c-40e6-aa30-ede0844470e8\") " pod="openshift-monitoring/thanos-querier-7f7dd99658-8wp4p"
Jul 16 00:41:28.227731 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:28.227615    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/15d0414e-398c-40e6-aa30-ede0844470e8-metrics-client-ca\") pod \"thanos-querier-7f7dd99658-8wp4p\" (UID: \"15d0414e-398c-40e6-aa30-ede0844470e8\") " pod="openshift-monitoring/thanos-querier-7f7dd99658-8wp4p"
Jul 16 00:41:28.228809 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:28.228784    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/15d0414e-398c-40e6-aa30-ede0844470e8-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7f7dd99658-8wp4p\" (UID: \"15d0414e-398c-40e6-aa30-ede0844470e8\") " pod="openshift-monitoring/thanos-querier-7f7dd99658-8wp4p"
Jul 16 00:41:28.228981 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:28.228933    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/15d0414e-398c-40e6-aa30-ede0844470e8-secret-thanos-querier-tls\") pod \"thanos-querier-7f7dd99658-8wp4p\" (UID: \"15d0414e-398c-40e6-aa30-ede0844470e8\") " pod="openshift-monitoring/thanos-querier-7f7dd99658-8wp4p"
Jul 16 00:41:28.229086 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:28.229009    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/15d0414e-398c-40e6-aa30-ede0844470e8-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7f7dd99658-8wp4p\" (UID: \"15d0414e-398c-40e6-aa30-ede0844470e8\") " pod="openshift-monitoring/thanos-querier-7f7dd99658-8wp4p"
Jul 16 00:41:28.229185 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:28.229165    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/15d0414e-398c-40e6-aa30-ede0844470e8-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7f7dd99658-8wp4p\" (UID: \"15d0414e-398c-40e6-aa30-ede0844470e8\") " pod="openshift-monitoring/thanos-querier-7f7dd99658-8wp4p"
Jul 16 00:41:28.230088 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:28.230062    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/15d0414e-398c-40e6-aa30-ede0844470e8-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7f7dd99658-8wp4p\" (UID: \"15d0414e-398c-40e6-aa30-ede0844470e8\") " pod="openshift-monitoring/thanos-querier-7f7dd99658-8wp4p"
Jul 16 00:41:28.230508 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:28.230491    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/15d0414e-398c-40e6-aa30-ede0844470e8-secret-grpc-tls\") pod \"thanos-querier-7f7dd99658-8wp4p\" (UID: \"15d0414e-398c-40e6-aa30-ede0844470e8\") " pod="openshift-monitoring/thanos-querier-7f7dd99658-8wp4p"
Jul 16 00:41:28.235699 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:28.235659    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mt7qv\" (UniqueName: \"kubernetes.io/projected/15d0414e-398c-40e6-aa30-ede0844470e8-kube-api-access-mt7qv\") pod \"thanos-querier-7f7dd99658-8wp4p\" (UID: \"15d0414e-398c-40e6-aa30-ede0844470e8\") " pod="openshift-monitoring/thanos-querier-7f7dd99658-8wp4p"
Jul 16 00:41:28.310023 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:28.309999    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-9tmlj"
Jul 16 00:41:28.349376 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:28.349287    2563 generic.go:356] "Generic (PLEG): container finished" podID="634869d9-271a-4673-b367-e673b5e7b867" containerID="462fd1b33390ed764f008fae8efb2e01d3dd323e002d132a97864d5ba9a0e49a" exitCode=0
Jul 16 00:41:28.349523 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:28.349372    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gfrth" event={"ID":"634869d9-271a-4673-b367-e673b5e7b867","Type":"ContainerDied","Data":"462fd1b33390ed764f008fae8efb2e01d3dd323e002d132a97864d5ba9a0e49a"}
Jul 16 00:41:28.351988 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:28.351949    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-sxrn4" event={"ID":"13f8df0b-6a12-4dca-95f4-ca467c9aad65","Type":"ContainerStarted","Data":"fa0d0b4c1bdc905c24ef38a51d7dd9f0b448f895f4a65f510d63ef63ef53b8fc"}
Jul 16 00:41:28.362593 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:28.362576    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7f7dd99658-8wp4p"
Jul 16 00:41:28.397450 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:28.397401    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-sxrn4" podStartSLOduration=1.270898318 podStartE2EDuration="2.397385152s" totalImagesPullingTime="1.126486834s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 00:41:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 00:41:28.396315497 +0000 UTC m=+60.831577237" watchObservedRunningTime="2026-07-16 00:41:28.397385152 +0000 UTC m=+60.832646891"
Jul 16 00:41:28.500028 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:28.499968    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7f7dd99658-8wp4p"]
Jul 16 00:41:28.502272 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:41:28.502245    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15d0414e_398c_40e6_aa30_ede0844470e8.slice/crio-270f5026a3d91ea4271c19a6e84ad5ca9d707fe00ff3b67a98266126537c64ff WatchSource:0}: Error finding container 270f5026a3d91ea4271c19a6e84ad5ca9d707fe00ff3b67a98266126537c64ff: Status 404 returned error can't find the container with id 270f5026a3d91ea4271c19a6e84ad5ca9d707fe00ff3b67a98266126537c64ff
Jul 16 00:41:28.832781 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:28.832751    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6bcbf625-91d3-456a-aa3d-7875407ec889-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6bcbf625-91d3-456a-aa3d-7875407ec889\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 00:41:28.835206 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:28.835185    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6bcbf625-91d3-456a-aa3d-7875407ec889-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6bcbf625-91d3-456a-aa3d-7875407ec889\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 00:41:28.979506 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:28.979474    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-lqhwj\""
Jul 16 00:41:28.987818 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:28.987788    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 16 00:41:29.128973 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:29.128883    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 16 00:41:29.132362 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:41:29.132335    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bcbf625_91d3_456a_aa3d_7875407ec889.slice/crio-2dd2d1989e6e94e122528226d1d457026f8c35aaa94512edbaea00a96cd64908 WatchSource:0}: Error finding container 2dd2d1989e6e94e122528226d1d457026f8c35aaa94512edbaea00a96cd64908: Status 404 returned error can't find the container with id 2dd2d1989e6e94e122528226d1d457026f8c35aaa94512edbaea00a96cd64908
Jul 16 00:41:29.356359 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:29.356315    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6bcbf625-91d3-456a-aa3d-7875407ec889","Type":"ContainerStarted","Data":"2dd2d1989e6e94e122528226d1d457026f8c35aaa94512edbaea00a96cd64908"}
Jul 16 00:41:29.357705 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:29.357674    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7f7dd99658-8wp4p" event={"ID":"15d0414e-398c-40e6-aa30-ede0844470e8","Type":"ContainerStarted","Data":"270f5026a3d91ea4271c19a6e84ad5ca9d707fe00ff3b67a98266126537c64ff"}
Jul 16 00:41:29.360041 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:29.360013    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gfrth" event={"ID":"634869d9-271a-4673-b367-e673b5e7b867","Type":"ContainerStarted","Data":"55673109d9178dc38f2210a61b120d81c9ddab0bb3e8a5198e95d41c5225e31b"}
Jul 16 00:41:29.360147 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:29.360065    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gfrth" event={"ID":"634869d9-271a-4673-b367-e673b5e7b867","Type":"ContainerStarted","Data":"d094ab9c38a5cda0dfe7823ac8818ad1ef75a6cfb5a4d32db25a5481af861ed0"}
Jul 16 00:41:29.380348 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:29.380244    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-gfrth" podStartSLOduration=1.877234879 podStartE2EDuration="3.380226143s" totalImagesPullingTime="1.434602433s" totalInitContainerRuntime="68.388831ms" isStatefulPod=true podCreationTimestamp="2026-07-16 00:41:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 00:41:29.379561921 +0000 UTC m=+61.814823661" watchObservedRunningTime="2026-07-16 00:41:29.380226143 +0000 UTC m=+61.815487882"
Jul 16 00:41:30.255053 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:30.255026    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bn5hr"
Jul 16 00:41:30.347894 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:30.347846    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-67fddbd646-n6lrk"]
Jul 16 00:41:30.350922 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:30.350903    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-67fddbd646-n6lrk"
Jul 16 00:41:30.353755 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:30.353734    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-x8vgx\""
Jul 16 00:41:30.353891 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:30.353832    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Jul 16 00:41:30.353891 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:30.353852    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Jul 16 00:41:30.354843 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:30.354614    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-d6q8ps92hjc4k\""
Jul 16 00:41:30.354843 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:30.354647    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Jul 16 00:41:30.354843 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:30.354749    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 16 00:41:30.363578 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:30.363557    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-67fddbd646-n6lrk"]
Jul 16 00:41:30.449319 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:30.449262    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/151b0be5-d373-4629-89b5-cf407bdab706-client-ca-bundle\") pod \"metrics-server-67fddbd646-n6lrk\" (UID: \"151b0be5-d373-4629-89b5-cf407bdab706\") " pod="openshift-monitoring/metrics-server-67fddbd646-n6lrk"
Jul 16 00:41:30.449509 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:30.449348    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/151b0be5-d373-4629-89b5-cf407bdab706-secret-metrics-server-client-certs\") pod \"metrics-server-67fddbd646-n6lrk\" (UID: \"151b0be5-d373-4629-89b5-cf407bdab706\") " pod="openshift-monitoring/metrics-server-67fddbd646-n6lrk"
Jul 16 00:41:30.449509 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:30.449378    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/151b0be5-d373-4629-89b5-cf407bdab706-audit-log\") pod \"metrics-server-67fddbd646-n6lrk\" (UID: \"151b0be5-d373-4629-89b5-cf407bdab706\") " pod="openshift-monitoring/metrics-server-67fddbd646-n6lrk"
Jul 16 00:41:30.449509 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:30.449423    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/151b0be5-d373-4629-89b5-cf407bdab706-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-67fddbd646-n6lrk\" (UID: \"151b0be5-d373-4629-89b5-cf407bdab706\") " pod="openshift-monitoring/metrics-server-67fddbd646-n6lrk"
Jul 16 00:41:30.449509 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:30.449461    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/151b0be5-d373-4629-89b5-cf407bdab706-secret-metrics-server-tls\") pod \"metrics-server-67fddbd646-n6lrk\" (UID: \"151b0be5-d373-4629-89b5-cf407bdab706\") " pod="openshift-monitoring/metrics-server-67fddbd646-n6lrk"
Jul 16 00:41:30.449509 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:30.449485    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/151b0be5-d373-4629-89b5-cf407bdab706-metrics-server-audit-profiles\") pod \"metrics-server-67fddbd646-n6lrk\" (UID: \"151b0be5-d373-4629-89b5-cf407bdab706\") " pod="openshift-monitoring/metrics-server-67fddbd646-n6lrk"
Jul 16 00:41:30.449509 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:30.449507    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-454s2\" (UniqueName: \"kubernetes.io/projected/151b0be5-d373-4629-89b5-cf407bdab706-kube-api-access-454s2\") pod \"metrics-server-67fddbd646-n6lrk\" (UID: \"151b0be5-d373-4629-89b5-cf407bdab706\") " pod="openshift-monitoring/metrics-server-67fddbd646-n6lrk"
Jul 16 00:41:30.550133 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:30.550045    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/151b0be5-d373-4629-89b5-cf407bdab706-secret-metrics-server-tls\") pod \"metrics-server-67fddbd646-n6lrk\" (UID: \"151b0be5-d373-4629-89b5-cf407bdab706\") " pod="openshift-monitoring/metrics-server-67fddbd646-n6lrk"
Jul 16 00:41:30.550133 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:30.550082    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/151b0be5-d373-4629-89b5-cf407bdab706-metrics-server-audit-profiles\") pod \"metrics-server-67fddbd646-n6lrk\" (UID: \"151b0be5-d373-4629-89b5-cf407bdab706\") " pod="openshift-monitoring/metrics-server-67fddbd646-n6lrk"
Jul 16 00:41:30.550133 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:30.550106    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-454s2\" (UniqueName: \"kubernetes.io/projected/151b0be5-d373-4629-89b5-cf407bdab706-kube-api-access-454s2\") pod \"metrics-server-67fddbd646-n6lrk\" (UID: \"151b0be5-d373-4629-89b5-cf407bdab706\") " pod="openshift-monitoring/metrics-server-67fddbd646-n6lrk"
Jul 16 00:41:30.550407 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:30.550184    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/151b0be5-d373-4629-89b5-cf407bdab706-client-ca-bundle\") pod \"metrics-server-67fddbd646-n6lrk\" (UID: \"151b0be5-d373-4629-89b5-cf407bdab706\") " pod="openshift-monitoring/metrics-server-67fddbd646-n6lrk"
Jul 16 00:41:30.550407 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:30.550231    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/151b0be5-d373-4629-89b5-cf407bdab706-secret-metrics-server-client-certs\") pod \"metrics-server-67fddbd646-n6lrk\" (UID: \"151b0be5-d373-4629-89b5-cf407bdab706\") " pod="openshift-monitoring/metrics-server-67fddbd646-n6lrk"
Jul 16 00:41:30.550407 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:30.550261    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/151b0be5-d373-4629-89b5-cf407bdab706-audit-log\") pod \"metrics-server-67fddbd646-n6lrk\" (UID: \"151b0be5-d373-4629-89b5-cf407bdab706\") " pod="openshift-monitoring/metrics-server-67fddbd646-n6lrk"
Jul 16 00:41:30.550407 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:30.550310    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/151b0be5-d373-4629-89b5-cf407bdab706-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-67fddbd646-n6lrk\" (UID: \"151b0be5-d373-4629-89b5-cf407bdab706\") " pod="openshift-monitoring/metrics-server-67fddbd646-n6lrk"
Jul 16 00:41:30.550981 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:30.550913    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/151b0be5-d373-4629-89b5-cf407bdab706-audit-log\") pod \"metrics-server-67fddbd646-n6lrk\" (UID: \"151b0be5-d373-4629-89b5-cf407bdab706\") " pod="openshift-monitoring/metrics-server-67fddbd646-n6lrk"
Jul 16 00:41:30.551111 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:30.550999    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/151b0be5-d373-4629-89b5-cf407bdab706-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-67fddbd646-n6lrk\" (UID: \"151b0be5-d373-4629-89b5-cf407bdab706\") " pod="openshift-monitoring/metrics-server-67fddbd646-n6lrk"
Jul 16 00:41:30.551184 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:30.551172    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/151b0be5-d373-4629-89b5-cf407bdab706-metrics-server-audit-profiles\") pod \"metrics-server-67fddbd646-n6lrk\" (UID: \"151b0be5-d373-4629-89b5-cf407bdab706\") " pod="openshift-monitoring/metrics-server-67fddbd646-n6lrk"
Jul 16 00:41:30.553078 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:30.553049    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/151b0be5-d373-4629-89b5-cf407bdab706-secret-metrics-server-client-certs\") pod \"metrics-server-67fddbd646-n6lrk\" (UID: \"151b0be5-d373-4629-89b5-cf407bdab706\") " pod="openshift-monitoring/metrics-server-67fddbd646-n6lrk"
Jul 16 00:41:30.553177 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:30.553106    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/151b0be5-d373-4629-89b5-cf407bdab706-secret-metrics-server-tls\") pod \"metrics-server-67fddbd646-n6lrk\" (UID: \"151b0be5-d373-4629-89b5-cf407bdab706\") " pod="openshift-monitoring/metrics-server-67fddbd646-n6lrk"
Jul 16 00:41:30.553232 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:30.553201    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/151b0be5-d373-4629-89b5-cf407bdab706-client-ca-bundle\") pod \"metrics-server-67fddbd646-n6lrk\" (UID: \"151b0be5-d373-4629-89b5-cf407bdab706\") " pod="openshift-monitoring/metrics-server-67fddbd646-n6lrk"
Jul 16 00:41:30.558520 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:30.558500    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-454s2\" (UniqueName: \"kubernetes.io/projected/151b0be5-d373-4629-89b5-cf407bdab706-kube-api-access-454s2\") pod \"metrics-server-67fddbd646-n6lrk\" (UID: \"151b0be5-d373-4629-89b5-cf407bdab706\") " pod="openshift-monitoring/metrics-server-67fddbd646-n6lrk"
Jul 16 00:41:30.662061 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:30.662021    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-67fddbd646-n6lrk"
Jul 16 00:41:30.824369 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:30.824333    2563 patch_prober.go:28] interesting pod/image-registry-5fd576b4b9-pwb8z container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 16 00:41:30.824520 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:30.824388    2563 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-5fd576b4b9-pwb8z" podUID="4562419f-a9c7-499b-b9c3-c1d4ccad25d1" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 16 00:41:31.070751 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:31.070723    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-67fddbd646-n6lrk"]
Jul 16 00:41:31.106141 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:41:31.106115    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod151b0be5_d373_4629_89b5_cf407bdab706.slice/crio-ef1cd76909b033f18163715f5fe30cb72805353763be0426a59e4f8fb9331908 WatchSource:0}: Error finding container ef1cd76909b033f18163715f5fe30cb72805353763be0426a59e4f8fb9331908: Status 404 returned error can't find the container with id ef1cd76909b033f18163715f5fe30cb72805353763be0426a59e4f8fb9331908
Jul 16 00:41:31.369527 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:31.369446    2563 generic.go:356] "Generic (PLEG): container finished" podID="6bcbf625-91d3-456a-aa3d-7875407ec889" containerID="c5f3aad8a4227c6ab9708048d734979057f9ff837fab4374ac3edd2844ab5fb4" exitCode=0
Jul 16 00:41:31.369966 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:31.369524    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6bcbf625-91d3-456a-aa3d-7875407ec889","Type":"ContainerDied","Data":"c5f3aad8a4227c6ab9708048d734979057f9ff837fab4374ac3edd2844ab5fb4"}
Jul 16 00:41:31.371901 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:31.371873    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7f7dd99658-8wp4p" event={"ID":"15d0414e-398c-40e6-aa30-ede0844470e8","Type":"ContainerStarted","Data":"d118a07dd04a8f150a049ffd3e58aa0c3c951e25470eb38dda0547a52fc33e47"}
Jul 16 00:41:31.372010 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:31.371934    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7f7dd99658-8wp4p" event={"ID":"15d0414e-398c-40e6-aa30-ede0844470e8","Type":"ContainerStarted","Data":"f4eabda747ebe54b60c2a2e4a90774300f42d8eec3e3377119db2f53da3b6a52"}
Jul 16 00:41:31.372010 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:31.371976    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7f7dd99658-8wp4p" event={"ID":"15d0414e-398c-40e6-aa30-ede0844470e8","Type":"ContainerStarted","Data":"29853c4a7bd37ab7e98d62f3dd37629af7fa751610ff3042667b26089b057a1f"}
Jul 16 00:41:31.373098 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:31.373069    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-67fddbd646-n6lrk" event={"ID":"151b0be5-d373-4629-89b5-cf407bdab706","Type":"ContainerStarted","Data":"ef1cd76909b033f18163715f5fe30cb72805353763be0426a59e4f8fb9331908"}
Jul 16 00:41:32.290991 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:32.290961    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5fd576b4b9-pwb8z"
Jul 16 00:41:32.408925 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:32.408895    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 16 00:41:32.413387 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:32.413366    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 00:41:32.416001 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:32.415979    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 16 00:41:32.416280 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:32.416184    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 16 00:41:32.416410 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:32.416358    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 16 00:41:32.416696 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:32.416678    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 16 00:41:32.416787 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:32.416772    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 16 00:41:32.416856 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:32.416678    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 16 00:41:32.417107 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:32.417085    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-3oml8al8okbbk\""
Jul 16 00:41:32.418311 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:32.418288    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 16 00:41:32.419080 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:32.419055    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-x4z5j\""
Jul 16 00:41:32.419188 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:32.419118    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 16 00:41:32.419252 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:32.419197    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 16 00:41:32.419428 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:32.419411    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 16 00:41:32.419558 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:32.419544    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 16 00:41:32.420142 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:32.420118    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 16 00:41:32.423570 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:32.423104    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 16 00:41:32.430180 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:32.429420    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 16 00:41:32.433091 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:32.433073    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 16 00:41:32.568340 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:32.568262    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d52ea21c-7d16-4007-a680-b67f3cfbcdc6-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d52ea21c-7d16-4007-a680-b67f3cfbcdc6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 00:41:32.568340 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:32.568331    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d52ea21c-7d16-4007-a680-b67f3cfbcdc6-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d52ea21c-7d16-4007-a680-b67f3cfbcdc6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 00:41:32.568549 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:32.568403    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d52ea21c-7d16-4007-a680-b67f3cfbcdc6-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d52ea21c-7d16-4007-a680-b67f3cfbcdc6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 00:41:32.568549 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:32.568443    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d52ea21c-7d16-4007-a680-b67f3cfbcdc6-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d52ea21c-7d16-4007-a680-b67f3cfbcdc6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 00:41:32.568644 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:32.568541    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d52ea21c-7d16-4007-a680-b67f3cfbcdc6-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d52ea21c-7d16-4007-a680-b67f3cfbcdc6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 00:41:32.568644 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:32.568580    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d52ea21c-7d16-4007-a680-b67f3cfbcdc6-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d52ea21c-7d16-4007-a680-b67f3cfbcdc6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 00:41:32.568644 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:32.568612    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d52ea21c-7d16-4007-a680-b67f3cfbcdc6-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d52ea21c-7d16-4007-a680-b67f3cfbcdc6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 00:41:32.568760 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:32.568653    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d52ea21c-7d16-4007-a680-b67f3cfbcdc6-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d52ea21c-7d16-4007-a680-b67f3cfbcdc6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 00:41:32.568760 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:32.568685    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d52ea21c-7d16-4007-a680-b67f3cfbcdc6-config-out\") pod \"prometheus-k8s-0\" (UID: \"d52ea21c-7d16-4007-a680-b67f3cfbcdc6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 00:41:32.568760 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:32.568709    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d52ea21c-7d16-4007-a680-b67f3cfbcdc6-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d52ea21c-7d16-4007-a680-b67f3cfbcdc6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 00:41:32.568760 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:32.568732    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d52ea21c-7d16-4007-a680-b67f3cfbcdc6-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d52ea21c-7d16-4007-a680-b67f3cfbcdc6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 00:41:32.568965 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:32.568801    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d52ea21c-7d16-4007-a680-b67f3cfbcdc6-web-config\") pod \"prometheus-k8s-0\" (UID: \"d52ea21c-7d16-4007-a680-b67f3cfbcdc6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 00:41:32.568965 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:32.568849    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d52ea21c-7d16-4007-a680-b67f3cfbcdc6-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d52ea21c-7d16-4007-a680-b67f3cfbcdc6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 00:41:32.568965 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:32.568890    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdcwd\" (UniqueName: \"kubernetes.io/projected/d52ea21c-7d16-4007-a680-b67f3cfbcdc6-kube-api-access-rdcwd\") pod \"prometheus-k8s-0\" (UID: \"d52ea21c-7d16-4007-a680-b67f3cfbcdc6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 00:41:32.568965 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:32.568925    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d52ea21c-7d16-4007-a680-b67f3cfbcdc6-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d52ea21c-7d16-4007-a680-b67f3cfbcdc6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 00:41:32.568965 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:32.568949    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d52ea21c-7d16-4007-a680-b67f3cfbcdc6-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d52ea21c-7d16-4007-a680-b67f3cfbcdc6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 00:41:32.569158 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:32.568976    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d52ea21c-7d16-4007-a680-b67f3cfbcdc6-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d52ea21c-7d16-4007-a680-b67f3cfbcdc6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 00:41:32.569158 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:32.569003    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d52ea21c-7d16-4007-a680-b67f3cfbcdc6-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d52ea21c-7d16-4007-a680-b67f3cfbcdc6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 00:41:32.569158 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:32.569028    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d52ea21c-7d16-4007-a680-b67f3cfbcdc6-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d52ea21c-7d16-4007-a680-b67f3cfbcdc6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 00:41:32.569158 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:32.569062    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/d52ea21c-7d16-4007-a680-b67f3cfbcdc6-config\") pod \"prometheus-k8s-0\" (UID: \"d52ea21c-7d16-4007-a680-b67f3cfbcdc6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 00:41:32.669937 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:32.669914    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/d52ea21c-7d16-4007-a680-b67f3cfbcdc6-config\") pod \"prometheus-k8s-0\" (UID: \"d52ea21c-7d16-4007-a680-b67f3cfbcdc6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 00:41:32.670060 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:32.669971    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d52ea21c-7d16-4007-a680-b67f3cfbcdc6-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d52ea21c-7d16-4007-a680-b67f3cfbcdc6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 00:41:32.670060 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:32.670024    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d52ea21c-7d16-4007-a680-b67f3cfbcdc6-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d52ea21c-7d16-4007-a680-b67f3cfbcdc6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 00:41:32.670167 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:32.670062    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d52ea21c-7d16-4007-a680-b67f3cfbcdc6-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d52ea21c-7d16-4007-a680-b67f3cfbcdc6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 00:41:32.670167 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:32.670094    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d52ea21c-7d16-4007-a680-b67f3cfbcdc6-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d52ea21c-7d16-4007-a680-b67f3cfbcdc6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 00:41:32.670167 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:32.670142    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d52ea21c-7d16-4007-a680-b67f3cfbcdc6-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d52ea21c-7d16-4007-a680-b67f3cfbcdc6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 00:41:32.670318 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:32.670170    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d52ea21c-7d16-4007-a680-b67f3cfbcdc6-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d52ea21c-7d16-4007-a680-b67f3cfbcdc6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 00:41:32.670318 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:32.670195    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d52ea21c-7d16-4007-a680-b67f3cfbcdc6-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d52ea21c-7d16-4007-a680-b67f3cfbcdc6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 00:41:32.670318 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:32.670223    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d52ea21c-7d16-4007-a680-b67f3cfbcdc6-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d52ea21c-7d16-4007-a680-b67f3cfbcdc6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 00:41:32.670318 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:32.670255    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d52ea21c-7d16-4007-a680-b67f3cfbcdc6-config-out\") pod \"prometheus-k8s-0\" (UID: \"d52ea21c-7d16-4007-a680-b67f3cfbcdc6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 00:41:32.670318 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:32.670280    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d52ea21c-7d16-4007-a680-b67f3cfbcdc6-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d52ea21c-7d16-4007-a680-b67f3cfbcdc6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 00:41:32.670318 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:32.670306    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d52ea21c-7d16-4007-a680-b67f3cfbcdc6-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d52ea21c-7d16-4007-a680-b67f3cfbcdc6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 00:41:32.670594 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:32.670339    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d52ea21c-7d16-4007-a680-b67f3cfbcdc6-web-config\") pod \"prometheus-k8s-0\" (UID: \"d52ea21c-7d16-4007-a680-b67f3cfbcdc6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 00:41:32.670594 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:32.670388    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d52ea21c-7d16-4007-a680-b67f3cfbcdc6-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d52ea21c-7d16-4007-a680-b67f3cfbcdc6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 00:41:32.670594 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:32.670415    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdcwd\" (UniqueName: \"kubernetes.io/projected/d52ea21c-7d16-4007-a680-b67f3cfbcdc6-kube-api-access-rdcwd\") pod \"prometheus-k8s-0\" (UID: \"d52ea21c-7d16-4007-a680-b67f3cfbcdc6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 00:41:32.670594 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:32.670448    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d52ea21c-7d16-4007-a680-b67f3cfbcdc6-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d52ea21c-7d16-4007-a680-b67f3cfbcdc6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 00:41:32.670594 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:32.670471    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d52ea21c-7d16-4007-a680-b67f3cfbcdc6-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d52ea21c-7d16-4007-a680-b67f3cfbcdc6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 00:41:32.670594 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:32.670498    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d52ea21c-7d16-4007-a680-b67f3cfbcdc6-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d52ea21c-7d16-4007-a680-b67f3cfbcdc6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 00:41:32.670594 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:32.670522    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d52ea21c-7d16-4007-a680-b67f3cfbcdc6-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d52ea21c-7d16-4007-a680-b67f3cfbcdc6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 00:41:32.670594 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:32.670548    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d52ea21c-7d16-4007-a680-b67f3cfbcdc6-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d52ea21c-7d16-4007-a680-b67f3cfbcdc6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 00:41:32.672438 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:32.671661    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d52ea21c-7d16-4007-a680-b67f3cfbcdc6-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d52ea21c-7d16-4007-a680-b67f3cfbcdc6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 00:41:32.672438 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:32.671849    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d52ea21c-7d16-4007-a680-b67f3cfbcdc6-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d52ea21c-7d16-4007-a680-b67f3cfbcdc6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 00:41:32.672438 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:32.671988    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d52ea21c-7d16-4007-a680-b67f3cfbcdc6-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d52ea21c-7d16-4007-a680-b67f3cfbcdc6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 00:41:32.672438 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:32.672179    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d52ea21c-7d16-4007-a680-b67f3cfbcdc6-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d52ea21c-7d16-4007-a680-b67f3cfbcdc6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 00:41:32.673230 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:32.672661    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d52ea21c-7d16-4007-a680-b67f3cfbcdc6-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d52ea21c-7d16-4007-a680-b67f3cfbcdc6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 00:41:32.673230 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:32.672988    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d52ea21c-7d16-4007-a680-b67f3cfbcdc6-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d52ea21c-7d16-4007-a680-b67f3cfbcdc6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 00:41:32.673230 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:32.673058    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d52ea21c-7d16-4007-a680-b67f3cfbcdc6-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d52ea21c-7d16-4007-a680-b67f3cfbcdc6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 00:41:32.674058 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:32.673820    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d52ea21c-7d16-4007-a680-b67f3cfbcdc6-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d52ea21c-7d16-4007-a680-b67f3cfbcdc6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 00:41:32.674242 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:32.674186    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d52ea21c-7d16-4007-a680-b67f3cfbcdc6-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d52ea21c-7d16-4007-a680-b67f3cfbcdc6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 00:41:32.674323 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:32.674299    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d52ea21c-7d16-4007-a680-b67f3cfbcdc6-web-config\") pod \"prometheus-k8s-0\" (UID: \"d52ea21c-7d16-4007-a680-b67f3cfbcdc6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 00:41:32.674784 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:32.674739    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d52ea21c-7d16-4007-a680-b67f3cfbcdc6-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d52ea21c-7d16-4007-a680-b67f3cfbcdc6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 00:41:32.675358 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:32.675333    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/d52ea21c-7d16-4007-a680-b67f3cfbcdc6-config\") pod \"prometheus-k8s-0\" (UID: \"d52ea21c-7d16-4007-a680-b67f3cfbcdc6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 00:41:32.676667 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:32.676614    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d52ea21c-7d16-4007-a680-b67f3cfbcdc6-config-out\") pod \"prometheus-k8s-0\" (UID: \"d52ea21c-7d16-4007-a680-b67f3cfbcdc6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 00:41:32.676774 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:32.676759    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d52ea21c-7d16-4007-a680-b67f3cfbcdc6-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d52ea21c-7d16-4007-a680-b67f3cfbcdc6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 00:41:32.676904 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:32.676805    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d52ea21c-7d16-4007-a680-b67f3cfbcdc6-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d52ea21c-7d16-4007-a680-b67f3cfbcdc6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 00:41:32.677204 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:32.677189    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d52ea21c-7d16-4007-a680-b67f3cfbcdc6-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d52ea21c-7d16-4007-a680-b67f3cfbcdc6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 00:41:32.677439 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:32.677410    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d52ea21c-7d16-4007-a680-b67f3cfbcdc6-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d52ea21c-7d16-4007-a680-b67f3cfbcdc6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 00:41:32.677851 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:32.677829    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d52ea21c-7d16-4007-a680-b67f3cfbcdc6-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d52ea21c-7d16-4007-a680-b67f3cfbcdc6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 00:41:32.678280 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:32.678261    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d52ea21c-7d16-4007-a680-b67f3cfbcdc6-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d52ea21c-7d16-4007-a680-b67f3cfbcdc6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 00:41:32.681246 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:32.681213    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdcwd\" (UniqueName: \"kubernetes.io/projected/d52ea21c-7d16-4007-a680-b67f3cfbcdc6-kube-api-access-rdcwd\") pod \"prometheus-k8s-0\" (UID: \"d52ea21c-7d16-4007-a680-b67f3cfbcdc6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 00:41:32.725267 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:32.725244    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 00:41:32.887567 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:32.887540    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 16 00:41:32.891358 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:41:32.891333    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd52ea21c_7d16_4007_a680_b67f3cfbcdc6.slice/crio-5db6ddf319439353abd512a2b30f96c96df3fe238004825352ac2409e0d2912b WatchSource:0}: Error finding container 5db6ddf319439353abd512a2b30f96c96df3fe238004825352ac2409e0d2912b: Status 404 returned error can't find the container with id 5db6ddf319439353abd512a2b30f96c96df3fe238004825352ac2409e0d2912b
Jul 16 00:41:33.382976 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:33.382940    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-67fddbd646-n6lrk" event={"ID":"151b0be5-d373-4629-89b5-cf407bdab706","Type":"ContainerStarted","Data":"2e303b7434bea25f9926505e72719bd85b17d8647bb3b93a06856f3b2fe3e664"}
Jul 16 00:41:33.384792 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:33.384759    2563 generic.go:356] "Generic (PLEG): container finished" podID="d52ea21c-7d16-4007-a680-b67f3cfbcdc6" containerID="602de56208d0126d303fe4020a5be610e107e35d39c452204f4aab86032c3710" exitCode=0
Jul 16 00:41:33.384945 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:33.384841    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d52ea21c-7d16-4007-a680-b67f3cfbcdc6","Type":"ContainerDied","Data":"602de56208d0126d303fe4020a5be610e107e35d39c452204f4aab86032c3710"}
Jul 16 00:41:33.384945 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:33.384935    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d52ea21c-7d16-4007-a680-b67f3cfbcdc6","Type":"ContainerStarted","Data":"5db6ddf319439353abd512a2b30f96c96df3fe238004825352ac2409e0d2912b"}
Jul 16 00:41:33.388968 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:33.388944    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7f7dd99658-8wp4p" event={"ID":"15d0414e-398c-40e6-aa30-ede0844470e8","Type":"ContainerStarted","Data":"ea9eb22486153ce4c24000d622a4140b623e76d63c7233fd791d88a94c65ea6a"}
Jul 16 00:41:33.389063 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:33.389016    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7f7dd99658-8wp4p" event={"ID":"15d0414e-398c-40e6-aa30-ede0844470e8","Type":"ContainerStarted","Data":"9809b7910003683ef6226f11d2f6205852a0483804e97357a53413d8edc46326"}
Jul 16 00:41:33.389063 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:33.389057    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7f7dd99658-8wp4p" event={"ID":"15d0414e-398c-40e6-aa30-ede0844470e8","Type":"ContainerStarted","Data":"608f1062471f0798d87813f9eb95a9a9aa4dad1667a11d21e45c7f75f413aedb"}
Jul 16 00:41:33.389217 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:33.389198    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-7f7dd99658-8wp4p"
Jul 16 00:41:33.401970 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:33.401921    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-67fddbd646-n6lrk" podStartSLOduration=1.857079563 podStartE2EDuration="3.401906474s" totalImagesPullingTime="1.544826911s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 00:41:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 00:41:33.400687882 +0000 UTC m=+65.835949621" watchObservedRunningTime="2026-07-16 00:41:33.401906474 +0000 UTC m=+65.837168209"
Jul 16 00:41:33.421968 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:33.421922    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-7f7dd99658-8wp4p" podStartSLOduration=1.576672942 podStartE2EDuration="5.421906761s" totalImagesPullingTime="3.845233819s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 00:41:28 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 00:41:33.421275383 +0000 UTC m=+65.856537121" watchObservedRunningTime="2026-07-16 00:41:33.421906761 +0000 UTC m=+65.857168499"
Jul 16 00:41:33.782924 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:33.782823    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/051e94d1-76fa-484b-b118-b29951f4c5ae-metrics-certs\") pod \"network-metrics-daemon-bkf2h\" (UID: \"051e94d1-76fa-484b-b118-b29951f4c5ae\") " pod="openshift-multus/network-metrics-daemon-bkf2h"
Jul 16 00:41:33.782924 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:33.782912    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/16a7fa19-0ce5-49ca-96ef-dde5ae694dc0-original-pull-secret\") pod \"global-pull-secret-syncer-hgcmx\" (UID: \"16a7fa19-0ce5-49ca-96ef-dde5ae694dc0\") " pod="kube-system/global-pull-secret-syncer-hgcmx"
Jul 16 00:41:33.785055 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:33.785025    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 16 00:41:33.785055 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:33.785049    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 16 00:41:33.795242 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:33.795216    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/051e94d1-76fa-484b-b118-b29951f4c5ae-metrics-certs\") pod \"network-metrics-daemon-bkf2h\" (UID: \"051e94d1-76fa-484b-b118-b29951f4c5ae\") " pod="openshift-multus/network-metrics-daemon-bkf2h"
Jul 16 00:41:33.795347 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:33.795260    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/16a7fa19-0ce5-49ca-96ef-dde5ae694dc0-original-pull-secret\") pod \"global-pull-secret-syncer-hgcmx\" (UID: \"16a7fa19-0ce5-49ca-96ef-dde5ae694dc0\") " pod="kube-system/global-pull-secret-syncer-hgcmx"
Jul 16 00:41:33.883489 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:33.883460    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7j7f\" (UniqueName: \"kubernetes.io/projected/1a9f4afb-53aa-401c-9a57-0cfc7e2f6af0-kube-api-access-t7j7f\") pod \"network-check-target-btw92\" (UID: \"1a9f4afb-53aa-401c-9a57-0cfc7e2f6af0\") " pod="openshift-network-diagnostics/network-check-target-btw92"
Jul 16 00:41:33.889323 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:33.889298    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 16 00:41:33.895829 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:33.895806    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 16 00:41:33.906136 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:33.906114    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7j7f\" (UniqueName: \"kubernetes.io/projected/1a9f4afb-53aa-401c-9a57-0cfc7e2f6af0-kube-api-access-t7j7f\") pod \"network-check-target-btw92\" (UID: \"1a9f4afb-53aa-401c-9a57-0cfc7e2f6af0\") " pod="openshift-network-diagnostics/network-check-target-btw92"
Jul 16 00:41:33.928448 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:33.928429    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-hn9ps\""
Jul 16 00:41:33.936428 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:33.936398    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-btw92"
Jul 16 00:41:33.938174 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:33.938156    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-vmvq6\""
Jul 16 00:41:33.943801 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:33.943776    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hgcmx"
Jul 16 00:41:33.946888 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:33.946859    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bkf2h"
Jul 16 00:41:34.108441 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:34.108413    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-btw92"]
Jul 16 00:41:34.111197 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:41:34.111170    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a9f4afb_53aa_401c_9a57_0cfc7e2f6af0.slice/crio-de850cd3cd6684858043de990a115388ac8f5100ebb6852b848e303e0899ff2e WatchSource:0}: Error finding container de850cd3cd6684858043de990a115388ac8f5100ebb6852b848e303e0899ff2e: Status 404 returned error can't find the container with id de850cd3cd6684858043de990a115388ac8f5100ebb6852b848e303e0899ff2e
Jul 16 00:41:34.193017 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:34.192206    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-hgcmx"]
Jul 16 00:41:34.201313 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:41:34.201267    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16a7fa19_0ce5_49ca_96ef_dde5ae694dc0.slice/crio-db58d3a7a2136ed3a34141727697eed2c7dd7a86706f59a93b260cc7f2362f9a WatchSource:0}: Error finding container db58d3a7a2136ed3a34141727697eed2c7dd7a86706f59a93b260cc7f2362f9a: Status 404 returned error can't find the container with id db58d3a7a2136ed3a34141727697eed2c7dd7a86706f59a93b260cc7f2362f9a
Jul 16 00:41:34.247658 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:34.247547    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-bkf2h"]
Jul 16 00:41:34.251478 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:41:34.251451    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod051e94d1_76fa_484b_b118_b29951f4c5ae.slice/crio-b7ecb7605acef8eb7445108306098418e49e180c8809ff9f8e17f5ee4f4bb103 WatchSource:0}: Error finding container b7ecb7605acef8eb7445108306098418e49e180c8809ff9f8e17f5ee4f4bb103: Status 404 returned error can't find the container with id b7ecb7605acef8eb7445108306098418e49e180c8809ff9f8e17f5ee4f4bb103
Jul 16 00:41:34.395944 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:34.395895    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6bcbf625-91d3-456a-aa3d-7875407ec889","Type":"ContainerStarted","Data":"84ded9942320f3bce1598daf535ff19752e53eba8918757af2a247d13d083fe7"}
Jul 16 00:41:34.396070 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:34.395983    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6bcbf625-91d3-456a-aa3d-7875407ec889","Type":"ContainerStarted","Data":"9f6bbec3801e4908c1a02578e25a7df518e0f65cf37f96d84c1667f00dcf3f50"}
Jul 16 00:41:34.396070 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:34.396025    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6bcbf625-91d3-456a-aa3d-7875407ec889","Type":"ContainerStarted","Data":"6555d628b99072b5653f5d59037110639add6840fd12b94532db476df6414e3c"}
Jul 16 00:41:34.397322 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:34.397263    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bkf2h" event={"ID":"051e94d1-76fa-484b-b118-b29951f4c5ae","Type":"ContainerStarted","Data":"b7ecb7605acef8eb7445108306098418e49e180c8809ff9f8e17f5ee4f4bb103"}
Jul 16 00:41:34.398699 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:34.398670    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-hgcmx" event={"ID":"16a7fa19-0ce5-49ca-96ef-dde5ae694dc0","Type":"ContainerStarted","Data":"db58d3a7a2136ed3a34141727697eed2c7dd7a86706f59a93b260cc7f2362f9a"}
Jul 16 00:41:34.400148 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:34.400104    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-btw92" event={"ID":"1a9f4afb-53aa-401c-9a57-0cfc7e2f6af0","Type":"ContainerStarted","Data":"de850cd3cd6684858043de990a115388ac8f5100ebb6852b848e303e0899ff2e"}
Jul 16 00:41:35.412039 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:35.412000    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6bcbf625-91d3-456a-aa3d-7875407ec889","Type":"ContainerStarted","Data":"cff855898631ed810a637abf06f68f3a573dd24b0ac0974f10e0f3a2b4b3d938"}
Jul 16 00:41:35.412502 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:35.412090    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6bcbf625-91d3-456a-aa3d-7875407ec889","Type":"ContainerStarted","Data":"2257af4c5ec5edf92cfcfa9f705bcbfc512acbab4fed10828e19bff5ae436a8a"}
Jul 16 00:41:35.412502 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:35.412133    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6bcbf625-91d3-456a-aa3d-7875407ec889","Type":"ContainerStarted","Data":"215ddd2c9eae1c3abd53ffda0538bbaa9c04c41efd643e1e6acaf88d5f527eba"}
Jul 16 00:41:35.442013 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:35.440526    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=4.106185431 podStartE2EDuration="8.440502695s" totalImagesPullingTime="4.312019881s" totalInitContainerRuntime="22.297383ms" isStatefulPod=false podCreationTimestamp="2026-07-16 00:41:27 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 00:41:35.439606655 +0000 UTC m=+67.874868394" watchObservedRunningTime="2026-07-16 00:41:35.440502695 +0000 UTC m=+67.875764434"
Jul 16 00:41:36.420991 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:36.420944    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bkf2h" event={"ID":"051e94d1-76fa-484b-b118-b29951f4c5ae","Type":"ContainerStarted","Data":"d0504890f2726e9457d313be85223f25f4d1650aa46ac61dbeb3b905ea432822"}
Jul 16 00:41:36.421422 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:36.421074    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bkf2h" event={"ID":"051e94d1-76fa-484b-b118-b29951f4c5ae","Type":"ContainerStarted","Data":"b3edf60f3813f91f68bc35084682c6e015e0b41726963f8d7d8b7372a48e98ed"}
Jul 16 00:41:36.439456 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:36.439406    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-bkf2h" podStartSLOduration=67.186788806 podStartE2EDuration="1m8.439388355s" totalImagesPullingTime="1.252599549s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 00:40:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 00:41:36.4393778 +0000 UTC m=+68.874639540" watchObservedRunningTime="2026-07-16 00:41:36.439388355 +0000 UTC m=+68.874650095"
Jul 16 00:41:39.406823 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:39.406790    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-7f7dd99658-8wp4p"
Jul 16 00:41:40.440222 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:40.440174    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d52ea21c-7d16-4007-a680-b67f3cfbcdc6","Type":"ContainerStarted","Data":"233220bbd44752a0b45a1b01321a7b1d68e89cf5b75590206313f0afa56cc21f"}
Jul 16 00:41:40.440721 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:40.440275    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d52ea21c-7d16-4007-a680-b67f3cfbcdc6","Type":"ContainerStarted","Data":"edfb98ca40b9ca5d4d12e1054bbb357a4bedad235fe36b73fb1b5cdebb18c1e2"}
Jul 16 00:41:40.440721 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:40.440319    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d52ea21c-7d16-4007-a680-b67f3cfbcdc6","Type":"ContainerStarted","Data":"73377ca2c9f5f98bdae8ac167ace8f28d9170705feff90bbff87dbf4058659ad"}
Jul 16 00:41:40.440721 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:40.440364    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d52ea21c-7d16-4007-a680-b67f3cfbcdc6","Type":"ContainerStarted","Data":"da5e0a6e27ccb609f6341ca1d7ae16b2c6b3054dc2a24bfc2145fe9ede975837"}
Jul 16 00:41:40.440721 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:40.440406    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d52ea21c-7d16-4007-a680-b67f3cfbcdc6","Type":"ContainerStarted","Data":"3b3d32f5e37e91811bbc20f3c1b425fa39f05f30266673ef09e240b586a20fd7"}
Jul 16 00:41:40.440721 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:40.440445    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d52ea21c-7d16-4007-a680-b67f3cfbcdc6","Type":"ContainerStarted","Data":"96c91fdd8e52ee998ebf81468edc2bb168994118fa5fa82ffe1361c2440d3d1e"}
Jul 16 00:41:40.441661 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:40.441633    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-hgcmx" event={"ID":"16a7fa19-0ce5-49ca-96ef-dde5ae694dc0","Type":"ContainerStarted","Data":"03449d9fb239f2bde925ff49b80001f8aca4d93a9d7498262ddcd911558e173c"}
Jul 16 00:41:40.443083 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:40.443062    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-btw92" event={"ID":"1a9f4afb-53aa-401c-9a57-0cfc7e2f6af0","Type":"ContainerStarted","Data":"101fe2c57aeb724cf906d9226c7bef89653989d457ca69a97dabf3545de7770e"}
Jul 16 00:41:40.443257 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:40.443243    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-btw92"
Jul 16 00:41:40.485726 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:40.485670    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.135751209 podStartE2EDuration="8.48565423s" totalImagesPullingTime="6.321465414s" totalInitContainerRuntime="28.437607ms" isStatefulPod=false podCreationTimestamp="2026-07-16 00:41:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 00:41:40.484055911 +0000 UTC m=+72.919317649" watchObservedRunningTime="2026-07-16 00:41:40.48565423 +0000 UTC m=+72.920915970"
Jul 16 00:41:40.504254 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:40.504208    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-hgcmx" podStartSLOduration=66.985819055 podStartE2EDuration="1m12.504192104s" totalImagesPullingTime="5.518373049s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 00:40:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 00:41:40.502910961 +0000 UTC m=+72.938172700" watchObservedRunningTime="2026-07-16 00:41:40.504192104 +0000 UTC m=+72.939453842"
Jul 16 00:41:40.520971 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:40.520921    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-btw92" podStartSLOduration=66.916020167 podStartE2EDuration="1m12.520899225s" totalImagesPullingTime="5.604879058s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 00:40:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 00:41:40.51994942 +0000 UTC m=+72.955211171" watchObservedRunningTime="2026-07-16 00:41:40.520899225 +0000 UTC m=+72.956160966"
Jul 16 00:41:42.726073 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:42.726033    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 00:41:50.663216 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:50.663088    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-67fddbd646-n6lrk"
Jul 16 00:41:50.663216 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:41:50.663157    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-67fddbd646-n6lrk"
Jul 16 00:42:10.668099 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:10.668068    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-67fddbd646-n6lrk"
Jul 16 00:42:10.671954 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:10.671928    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-67fddbd646-n6lrk"
Jul 16 00:42:11.450466 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:11.450430    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-btw92"
Jul 16 00:42:32.725876 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:32.725839    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 00:42:32.743981 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:32.743959    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 00:42:33.659106 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:33.659079    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 00:42:46.434090 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:46.434053    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 16 00:42:46.434674 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:46.434429    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="6bcbf625-91d3-456a-aa3d-7875407ec889" containerName="alertmanager" containerID="cri-o://6555d628b99072b5653f5d59037110639add6840fd12b94532db476df6414e3c" gracePeriod=120
Jul 16 00:42:46.434674 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:46.434466    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="6bcbf625-91d3-456a-aa3d-7875407ec889" containerName="kube-rbac-proxy-metric" containerID="cri-o://2257af4c5ec5edf92cfcfa9f705bcbfc512acbab4fed10828e19bff5ae436a8a" gracePeriod=120
Jul 16 00:42:46.434674 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:46.434484    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="6bcbf625-91d3-456a-aa3d-7875407ec889" containerName="kube-rbac-proxy-web" containerID="cri-o://84ded9942320f3bce1598daf535ff19752e53eba8918757af2a247d13d083fe7" gracePeriod=120
Jul 16 00:42:46.434674 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:46.434506    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="6bcbf625-91d3-456a-aa3d-7875407ec889" containerName="kube-rbac-proxy" containerID="cri-o://215ddd2c9eae1c3abd53ffda0538bbaa9c04c41efd643e1e6acaf88d5f527eba" gracePeriod=120
Jul 16 00:42:46.434674 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:46.434570    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="6bcbf625-91d3-456a-aa3d-7875407ec889" containerName="config-reloader" containerID="cri-o://9f6bbec3801e4908c1a02578e25a7df518e0f65cf37f96d84c1667f00dcf3f50" gracePeriod=120
Jul 16 00:42:46.434674 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:46.434587    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="6bcbf625-91d3-456a-aa3d-7875407ec889" containerName="prom-label-proxy" containerID="cri-o://cff855898631ed810a637abf06f68f3a573dd24b0ac0974f10e0f3a2b4b3d938" gracePeriod=120
Jul 16 00:42:46.695980 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:46.695893    2563 generic.go:356] "Generic (PLEG): container finished" podID="6bcbf625-91d3-456a-aa3d-7875407ec889" containerID="cff855898631ed810a637abf06f68f3a573dd24b0ac0974f10e0f3a2b4b3d938" exitCode=0
Jul 16 00:42:46.695980 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:46.695921    2563 generic.go:356] "Generic (PLEG): container finished" podID="6bcbf625-91d3-456a-aa3d-7875407ec889" containerID="2257af4c5ec5edf92cfcfa9f705bcbfc512acbab4fed10828e19bff5ae436a8a" exitCode=0
Jul 16 00:42:46.695980 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:46.695927    2563 generic.go:356] "Generic (PLEG): container finished" podID="6bcbf625-91d3-456a-aa3d-7875407ec889" containerID="215ddd2c9eae1c3abd53ffda0538bbaa9c04c41efd643e1e6acaf88d5f527eba" exitCode=0
Jul 16 00:42:46.695980 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:46.695933    2563 generic.go:356] "Generic (PLEG): container finished" podID="6bcbf625-91d3-456a-aa3d-7875407ec889" containerID="9f6bbec3801e4908c1a02578e25a7df518e0f65cf37f96d84c1667f00dcf3f50" exitCode=0
Jul 16 00:42:46.695980 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:46.695941    2563 generic.go:356] "Generic (PLEG): container finished" podID="6bcbf625-91d3-456a-aa3d-7875407ec889" containerID="6555d628b99072b5653f5d59037110639add6840fd12b94532db476df6414e3c" exitCode=0
Jul 16 00:42:46.695980 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:46.695966    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6bcbf625-91d3-456a-aa3d-7875407ec889","Type":"ContainerDied","Data":"cff855898631ed810a637abf06f68f3a573dd24b0ac0974f10e0f3a2b4b3d938"}
Jul 16 00:42:46.696264 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:46.696027    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6bcbf625-91d3-456a-aa3d-7875407ec889","Type":"ContainerDied","Data":"2257af4c5ec5edf92cfcfa9f705bcbfc512acbab4fed10828e19bff5ae436a8a"}
Jul 16 00:42:46.696264 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:46.696053    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6bcbf625-91d3-456a-aa3d-7875407ec889","Type":"ContainerDied","Data":"215ddd2c9eae1c3abd53ffda0538bbaa9c04c41efd643e1e6acaf88d5f527eba"}
Jul 16 00:42:46.696264 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:46.696076    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6bcbf625-91d3-456a-aa3d-7875407ec889","Type":"ContainerDied","Data":"9f6bbec3801e4908c1a02578e25a7df518e0f65cf37f96d84c1667f00dcf3f50"}
Jul 16 00:42:46.696264 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:46.696098    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6bcbf625-91d3-456a-aa3d-7875407ec889","Type":"ContainerDied","Data":"6555d628b99072b5653f5d59037110639add6840fd12b94532db476df6414e3c"}
Jul 16 00:42:47.677980 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:47.677952    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 16 00:42:47.702710 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:47.702682    2563 generic.go:356] "Generic (PLEG): container finished" podID="6bcbf625-91d3-456a-aa3d-7875407ec889" containerID="84ded9942320f3bce1598daf535ff19752e53eba8918757af2a247d13d083fe7" exitCode=0
Jul 16 00:42:47.702845 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:47.702729    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6bcbf625-91d3-456a-aa3d-7875407ec889","Type":"ContainerDied","Data":"84ded9942320f3bce1598daf535ff19752e53eba8918757af2a247d13d083fe7"}
Jul 16 00:42:47.702845 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:47.702774    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 16 00:42:47.702845 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:47.702798    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6bcbf625-91d3-456a-aa3d-7875407ec889","Type":"ContainerDied","Data":"2dd2d1989e6e94e122528226d1d457026f8c35aaa94512edbaea00a96cd64908"}
Jul 16 00:42:47.702967 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:47.702847    2563 scope.go:117] "RemoveContainer" containerID="cff855898631ed810a637abf06f68f3a573dd24b0ac0974f10e0f3a2b4b3d938"
Jul 16 00:42:47.709985 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:47.709968    2563 scope.go:117] "RemoveContainer" containerID="2257af4c5ec5edf92cfcfa9f705bcbfc512acbab4fed10828e19bff5ae436a8a"
Jul 16 00:42:47.721102 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:47.720475    2563 scope.go:117] "RemoveContainer" containerID="215ddd2c9eae1c3abd53ffda0538bbaa9c04c41efd643e1e6acaf88d5f527eba"
Jul 16 00:42:47.730146 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:47.730118    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6bcbf625-91d3-456a-aa3d-7875407ec889-web-config\") pod \"6bcbf625-91d3-456a-aa3d-7875407ec889\" (UID: \"6bcbf625-91d3-456a-aa3d-7875407ec889\") "
Jul 16 00:42:47.730267 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:47.730160    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6bcbf625-91d3-456a-aa3d-7875407ec889-tls-assets\") pod \"6bcbf625-91d3-456a-aa3d-7875407ec889\" (UID: \"6bcbf625-91d3-456a-aa3d-7875407ec889\") "
Jul 16 00:42:47.730267 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:47.730193    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6bcbf625-91d3-456a-aa3d-7875407ec889-cluster-tls-config\") pod \"6bcbf625-91d3-456a-aa3d-7875407ec889\" (UID: \"6bcbf625-91d3-456a-aa3d-7875407ec889\") "
Jul 16 00:42:47.730267 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:47.730220    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6bcbf625-91d3-456a-aa3d-7875407ec889-secret-alertmanager-kube-rbac-proxy\") pod \"6bcbf625-91d3-456a-aa3d-7875407ec889\" (UID: \"6bcbf625-91d3-456a-aa3d-7875407ec889\") "
Jul 16 00:42:47.730267 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:47.730249    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dq97l\" (UniqueName: \"kubernetes.io/projected/6bcbf625-91d3-456a-aa3d-7875407ec889-kube-api-access-dq97l\") pod \"6bcbf625-91d3-456a-aa3d-7875407ec889\" (UID: \"6bcbf625-91d3-456a-aa3d-7875407ec889\") "
Jul 16 00:42:47.730484 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:47.730280    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6bcbf625-91d3-456a-aa3d-7875407ec889-alertmanager-trusted-ca-bundle\") pod \"6bcbf625-91d3-456a-aa3d-7875407ec889\" (UID: \"6bcbf625-91d3-456a-aa3d-7875407ec889\") "
Jul 16 00:42:47.730484 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:47.730346    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6bcbf625-91d3-456a-aa3d-7875407ec889-alertmanager-main-db\") pod \"6bcbf625-91d3-456a-aa3d-7875407ec889\" (UID: \"6bcbf625-91d3-456a-aa3d-7875407ec889\") "
Jul 16 00:42:47.730484 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:47.730372    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6bcbf625-91d3-456a-aa3d-7875407ec889-secret-alertmanager-kube-rbac-proxy-web\") pod \"6bcbf625-91d3-456a-aa3d-7875407ec889\" (UID: \"6bcbf625-91d3-456a-aa3d-7875407ec889\") "
Jul 16 00:42:47.730484 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:47.730410    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6bcbf625-91d3-456a-aa3d-7875407ec889-secret-alertmanager-kube-rbac-proxy-metric\") pod \"6bcbf625-91d3-456a-aa3d-7875407ec889\" (UID: \"6bcbf625-91d3-456a-aa3d-7875407ec889\") "
Jul 16 00:42:47.730484 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:47.730449    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6bcbf625-91d3-456a-aa3d-7875407ec889-secret-alertmanager-main-tls\") pod \"6bcbf625-91d3-456a-aa3d-7875407ec889\" (UID: \"6bcbf625-91d3-456a-aa3d-7875407ec889\") "
Jul 16 00:42:47.730728 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:47.730490    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6bcbf625-91d3-456a-aa3d-7875407ec889-metrics-client-ca\") pod \"6bcbf625-91d3-456a-aa3d-7875407ec889\" (UID: \"6bcbf625-91d3-456a-aa3d-7875407ec889\") "
Jul 16 00:42:47.730728 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:47.730532    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6bcbf625-91d3-456a-aa3d-7875407ec889-config-out\") pod \"6bcbf625-91d3-456a-aa3d-7875407ec889\" (UID: \"6bcbf625-91d3-456a-aa3d-7875407ec889\") "
Jul 16 00:42:47.730728 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:47.730575    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6bcbf625-91d3-456a-aa3d-7875407ec889-config-volume\") pod \"6bcbf625-91d3-456a-aa3d-7875407ec889\" (UID: \"6bcbf625-91d3-456a-aa3d-7875407ec889\") "
Jul 16 00:42:47.730909 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:47.730855    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bcbf625-91d3-456a-aa3d-7875407ec889-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "6bcbf625-91d3-456a-aa3d-7875407ec889" (UID: "6bcbf625-91d3-456a-aa3d-7875407ec889"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 00:42:47.731840 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:47.731799    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bcbf625-91d3-456a-aa3d-7875407ec889-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "6bcbf625-91d3-456a-aa3d-7875407ec889" (UID: "6bcbf625-91d3-456a-aa3d-7875407ec889"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 00:42:47.734660 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:47.734632    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6bcbf625-91d3-456a-aa3d-7875407ec889-config-volume" (OuterVolumeSpecName: "config-volume") pod "6bcbf625-91d3-456a-aa3d-7875407ec889" (UID: "6bcbf625-91d3-456a-aa3d-7875407ec889"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 00:42:47.734847 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:47.734819    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bcbf625-91d3-456a-aa3d-7875407ec889-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "6bcbf625-91d3-456a-aa3d-7875407ec889" (UID: "6bcbf625-91d3-456a-aa3d-7875407ec889"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 00:42:47.735542 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:47.735018    2563 scope.go:117] "RemoveContainer" containerID="84ded9942320f3bce1598daf535ff19752e53eba8918757af2a247d13d083fe7"
Jul 16 00:42:47.739124 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:47.738857    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6bcbf625-91d3-456a-aa3d-7875407ec889-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "6bcbf625-91d3-456a-aa3d-7875407ec889" (UID: "6bcbf625-91d3-456a-aa3d-7875407ec889"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 00:42:47.739505 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:47.739471    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6bcbf625-91d3-456a-aa3d-7875407ec889-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "6bcbf625-91d3-456a-aa3d-7875407ec889" (UID: "6bcbf625-91d3-456a-aa3d-7875407ec889"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 00:42:47.739614 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:47.739558    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6bcbf625-91d3-456a-aa3d-7875407ec889-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "6bcbf625-91d3-456a-aa3d-7875407ec889" (UID: "6bcbf625-91d3-456a-aa3d-7875407ec889"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 00:42:47.739679 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:47.739657    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bcbf625-91d3-456a-aa3d-7875407ec889-config-out" (OuterVolumeSpecName: "config-out") pod "6bcbf625-91d3-456a-aa3d-7875407ec889" (UID: "6bcbf625-91d3-456a-aa3d-7875407ec889"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 00:42:47.740009 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:47.739973    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6bcbf625-91d3-456a-aa3d-7875407ec889-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "6bcbf625-91d3-456a-aa3d-7875407ec889" (UID: "6bcbf625-91d3-456a-aa3d-7875407ec889"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 00:42:47.740344 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:47.740320    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bcbf625-91d3-456a-aa3d-7875407ec889-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "6bcbf625-91d3-456a-aa3d-7875407ec889" (UID: "6bcbf625-91d3-456a-aa3d-7875407ec889"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 00:42:47.742174 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:47.742125    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bcbf625-91d3-456a-aa3d-7875407ec889-kube-api-access-dq97l" (OuterVolumeSpecName: "kube-api-access-dq97l") pod "6bcbf625-91d3-456a-aa3d-7875407ec889" (UID: "6bcbf625-91d3-456a-aa3d-7875407ec889"). InnerVolumeSpecName "kube-api-access-dq97l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 00:42:47.742481 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:47.742452    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6bcbf625-91d3-456a-aa3d-7875407ec889-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "6bcbf625-91d3-456a-aa3d-7875407ec889" (UID: "6bcbf625-91d3-456a-aa3d-7875407ec889"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 00:42:47.748623 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:47.748596    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6bcbf625-91d3-456a-aa3d-7875407ec889-web-config" (OuterVolumeSpecName: "web-config") pod "6bcbf625-91d3-456a-aa3d-7875407ec889" (UID: "6bcbf625-91d3-456a-aa3d-7875407ec889"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 00:42:47.756988 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:47.756958    2563 scope.go:117] "RemoveContainer" containerID="9f6bbec3801e4908c1a02578e25a7df518e0f65cf37f96d84c1667f00dcf3f50"
Jul 16 00:42:47.764120 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:47.764099    2563 scope.go:117] "RemoveContainer" containerID="6555d628b99072b5653f5d59037110639add6840fd12b94532db476df6414e3c"
Jul 16 00:42:47.770983 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:47.770947    2563 scope.go:117] "RemoveContainer" containerID="c5f3aad8a4227c6ab9708048d734979057f9ff837fab4374ac3edd2844ab5fb4"
Jul 16 00:42:47.777574 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:47.777554    2563 scope.go:117] "RemoveContainer" containerID="cff855898631ed810a637abf06f68f3a573dd24b0ac0974f10e0f3a2b4b3d938"
Jul 16 00:42:47.777826 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:42:47.777809    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cff855898631ed810a637abf06f68f3a573dd24b0ac0974f10e0f3a2b4b3d938\": container with ID starting with cff855898631ed810a637abf06f68f3a573dd24b0ac0974f10e0f3a2b4b3d938 not found: ID does not exist" containerID="cff855898631ed810a637abf06f68f3a573dd24b0ac0974f10e0f3a2b4b3d938"
Jul 16 00:42:47.777885 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:47.777835    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cff855898631ed810a637abf06f68f3a573dd24b0ac0974f10e0f3a2b4b3d938"} err="failed to get container status \"cff855898631ed810a637abf06f68f3a573dd24b0ac0974f10e0f3a2b4b3d938\": rpc error: code = NotFound desc = could not find container \"cff855898631ed810a637abf06f68f3a573dd24b0ac0974f10e0f3a2b4b3d938\": container with ID starting with cff855898631ed810a637abf06f68f3a573dd24b0ac0974f10e0f3a2b4b3d938 not found: ID does not exist"
Jul 16 00:42:47.777930 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:47.777887    2563 scope.go:117] "RemoveContainer" containerID="2257af4c5ec5edf92cfcfa9f705bcbfc512acbab4fed10828e19bff5ae436a8a"
Jul 16 00:42:47.778131 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:42:47.778108    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2257af4c5ec5edf92cfcfa9f705bcbfc512acbab4fed10828e19bff5ae436a8a\": container with ID starting with 2257af4c5ec5edf92cfcfa9f705bcbfc512acbab4fed10828e19bff5ae436a8a not found: ID does not exist" containerID="2257af4c5ec5edf92cfcfa9f705bcbfc512acbab4fed10828e19bff5ae436a8a"
Jul 16 00:42:47.778218 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:47.778134    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2257af4c5ec5edf92cfcfa9f705bcbfc512acbab4fed10828e19bff5ae436a8a"} err="failed to get container status \"2257af4c5ec5edf92cfcfa9f705bcbfc512acbab4fed10828e19bff5ae436a8a\": rpc error: code = NotFound desc = could not find container \"2257af4c5ec5edf92cfcfa9f705bcbfc512acbab4fed10828e19bff5ae436a8a\": container with ID starting with 2257af4c5ec5edf92cfcfa9f705bcbfc512acbab4fed10828e19bff5ae436a8a not found: ID does not exist"
Jul 16 00:42:47.778218 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:47.778152    2563 scope.go:117] "RemoveContainer" containerID="215ddd2c9eae1c3abd53ffda0538bbaa9c04c41efd643e1e6acaf88d5f527eba"
Jul 16 00:42:47.778398 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:42:47.778381    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"215ddd2c9eae1c3abd53ffda0538bbaa9c04c41efd643e1e6acaf88d5f527eba\": container with ID starting with 215ddd2c9eae1c3abd53ffda0538bbaa9c04c41efd643e1e6acaf88d5f527eba not found: ID does not exist" containerID="215ddd2c9eae1c3abd53ffda0538bbaa9c04c41efd643e1e6acaf88d5f527eba"
Jul 16 00:42:47.778439 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:47.778403    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"215ddd2c9eae1c3abd53ffda0538bbaa9c04c41efd643e1e6acaf88d5f527eba"} err="failed to get container status \"215ddd2c9eae1c3abd53ffda0538bbaa9c04c41efd643e1e6acaf88d5f527eba\": rpc error: code = NotFound desc = could not find container \"215ddd2c9eae1c3abd53ffda0538bbaa9c04c41efd643e1e6acaf88d5f527eba\": container with ID starting with 215ddd2c9eae1c3abd53ffda0538bbaa9c04c41efd643e1e6acaf88d5f527eba not found: ID does not exist"
Jul 16 00:42:47.778439 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:47.778418    2563 scope.go:117] "RemoveContainer" containerID="84ded9942320f3bce1598daf535ff19752e53eba8918757af2a247d13d083fe7"
Jul 16 00:42:47.778622 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:42:47.778607    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84ded9942320f3bce1598daf535ff19752e53eba8918757af2a247d13d083fe7\": container with ID starting with 84ded9942320f3bce1598daf535ff19752e53eba8918757af2a247d13d083fe7 not found: ID does not exist" containerID="84ded9942320f3bce1598daf535ff19752e53eba8918757af2a247d13d083fe7"
Jul 16 00:42:47.778659 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:47.778627    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84ded9942320f3bce1598daf535ff19752e53eba8918757af2a247d13d083fe7"} err="failed to get container status \"84ded9942320f3bce1598daf535ff19752e53eba8918757af2a247d13d083fe7\": rpc error: code = NotFound desc = could not find container \"84ded9942320f3bce1598daf535ff19752e53eba8918757af2a247d13d083fe7\": container with ID starting with 84ded9942320f3bce1598daf535ff19752e53eba8918757af2a247d13d083fe7 not found: ID does not exist"
Jul 16 00:42:47.778659 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:47.778640    2563 scope.go:117] "RemoveContainer" containerID="9f6bbec3801e4908c1a02578e25a7df518e0f65cf37f96d84c1667f00dcf3f50"
Jul 16 00:42:47.778834 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:42:47.778819    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f6bbec3801e4908c1a02578e25a7df518e0f65cf37f96d84c1667f00dcf3f50\": container with ID starting with 9f6bbec3801e4908c1a02578e25a7df518e0f65cf37f96d84c1667f00dcf3f50 not found: ID does not exist" containerID="9f6bbec3801e4908c1a02578e25a7df518e0f65cf37f96d84c1667f00dcf3f50"
Jul 16 00:42:47.778891 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:47.778838    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f6bbec3801e4908c1a02578e25a7df518e0f65cf37f96d84c1667f00dcf3f50"} err="failed to get container status \"9f6bbec3801e4908c1a02578e25a7df518e0f65cf37f96d84c1667f00dcf3f50\": rpc error: code = NotFound desc = could not find container \"9f6bbec3801e4908c1a02578e25a7df518e0f65cf37f96d84c1667f00dcf3f50\": container with ID starting with 9f6bbec3801e4908c1a02578e25a7df518e0f65cf37f96d84c1667f00dcf3f50 not found: ID does not exist"
Jul 16 00:42:47.778891 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:47.778849    2563 scope.go:117] "RemoveContainer" containerID="6555d628b99072b5653f5d59037110639add6840fd12b94532db476df6414e3c"
Jul 16 00:42:47.779096 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:42:47.779080    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6555d628b99072b5653f5d59037110639add6840fd12b94532db476df6414e3c\": container with ID starting with 6555d628b99072b5653f5d59037110639add6840fd12b94532db476df6414e3c not found: ID does not exist" containerID="6555d628b99072b5653f5d59037110639add6840fd12b94532db476df6414e3c"
Jul 16 00:42:47.779139 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:47.779101    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6555d628b99072b5653f5d59037110639add6840fd12b94532db476df6414e3c"} err="failed to get container status \"6555d628b99072b5653f5d59037110639add6840fd12b94532db476df6414e3c\": rpc error: code = NotFound desc = could not find container \"6555d628b99072b5653f5d59037110639add6840fd12b94532db476df6414e3c\": container with ID starting with 6555d628b99072b5653f5d59037110639add6840fd12b94532db476df6414e3c not found: ID does not exist"
Jul 16 00:42:47.779139 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:47.779114    2563 scope.go:117] "RemoveContainer" containerID="c5f3aad8a4227c6ab9708048d734979057f9ff837fab4374ac3edd2844ab5fb4"
Jul 16 00:42:47.779348 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:42:47.779333    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5f3aad8a4227c6ab9708048d734979057f9ff837fab4374ac3edd2844ab5fb4\": container with ID starting with c5f3aad8a4227c6ab9708048d734979057f9ff837fab4374ac3edd2844ab5fb4 not found: ID does not exist" containerID="c5f3aad8a4227c6ab9708048d734979057f9ff837fab4374ac3edd2844ab5fb4"
Jul 16 00:42:47.779390 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:47.779350    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5f3aad8a4227c6ab9708048d734979057f9ff837fab4374ac3edd2844ab5fb4"} err="failed to get container status \"c5f3aad8a4227c6ab9708048d734979057f9ff837fab4374ac3edd2844ab5fb4\": rpc error: code = NotFound desc = could not find container \"c5f3aad8a4227c6ab9708048d734979057f9ff837fab4374ac3edd2844ab5fb4\": container with ID starting with c5f3aad8a4227c6ab9708048d734979057f9ff837fab4374ac3edd2844ab5fb4 not found: ID does not exist"
Jul 16 00:42:47.831694 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:47.831659    2563 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6bcbf625-91d3-456a-aa3d-7875407ec889-secret-alertmanager-main-tls\") on node \"ip-10-0-143-65.ec2.internal\" DevicePath \"\""
Jul 16 00:42:47.831694 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:47.831688    2563 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6bcbf625-91d3-456a-aa3d-7875407ec889-metrics-client-ca\") on node \"ip-10-0-143-65.ec2.internal\" DevicePath \"\""
Jul 16 00:42:47.831694 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:47.831701    2563 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6bcbf625-91d3-456a-aa3d-7875407ec889-config-out\") on node \"ip-10-0-143-65.ec2.internal\" DevicePath \"\""
Jul 16 00:42:47.831952 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:47.831709    2563 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6bcbf625-91d3-456a-aa3d-7875407ec889-config-volume\") on node \"ip-10-0-143-65.ec2.internal\" DevicePath \"\""
Jul 16 00:42:47.831952 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:47.831720    2563 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6bcbf625-91d3-456a-aa3d-7875407ec889-web-config\") on node \"ip-10-0-143-65.ec2.internal\" DevicePath \"\""
Jul 16 00:42:47.831952 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:47.831727    2563 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6bcbf625-91d3-456a-aa3d-7875407ec889-tls-assets\") on node \"ip-10-0-143-65.ec2.internal\" DevicePath \"\""
Jul 16 00:42:47.831952 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:47.831734    2563 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6bcbf625-91d3-456a-aa3d-7875407ec889-cluster-tls-config\") on node \"ip-10-0-143-65.ec2.internal\" DevicePath \"\""
Jul 16 00:42:47.831952 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:47.831743    2563 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6bcbf625-91d3-456a-aa3d-7875407ec889-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-143-65.ec2.internal\" DevicePath \"\""
Jul 16 00:42:47.831952 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:47.831752    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dq97l\" (UniqueName: \"kubernetes.io/projected/6bcbf625-91d3-456a-aa3d-7875407ec889-kube-api-access-dq97l\") on node \"ip-10-0-143-65.ec2.internal\" DevicePath \"\""
Jul 16 00:42:47.831952 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:47.831762    2563 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6bcbf625-91d3-456a-aa3d-7875407ec889-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-143-65.ec2.internal\" DevicePath \"\""
Jul 16 00:42:47.831952 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:47.831771    2563 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6bcbf625-91d3-456a-aa3d-7875407ec889-alertmanager-main-db\") on node \"ip-10-0-143-65.ec2.internal\" DevicePath \"\""
Jul 16 00:42:47.831952 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:47.831780    2563 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6bcbf625-91d3-456a-aa3d-7875407ec889-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-143-65.ec2.internal\" DevicePath \"\""
Jul 16 00:42:47.831952 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:47.831789    2563 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6bcbf625-91d3-456a-aa3d-7875407ec889-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-143-65.ec2.internal\" DevicePath \"\""
Jul 16 00:42:48.030392 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:48.030320    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 16 00:42:48.040065 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:48.040038    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 16 00:42:48.077987 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:48.077958    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 16 00:42:48.078238 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:48.078225    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6bcbf625-91d3-456a-aa3d-7875407ec889" containerName="init-config-reloader"
Jul 16 00:42:48.078280 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:48.078240    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6bcbf625-91d3-456a-aa3d-7875407ec889" containerName="init-config-reloader"
Jul 16 00:42:48.078313 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:48.078283    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6bcbf625-91d3-456a-aa3d-7875407ec889" containerName="alertmanager"
Jul 16 00:42:48.078313 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:48.078292    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6bcbf625-91d3-456a-aa3d-7875407ec889" containerName="alertmanager"
Jul 16 00:42:48.078313 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:48.078302    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6bcbf625-91d3-456a-aa3d-7875407ec889" containerName="prom-label-proxy"
Jul 16 00:42:48.078313 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:48.078307    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6bcbf625-91d3-456a-aa3d-7875407ec889" containerName="prom-label-proxy"
Jul 16 00:42:48.078454 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:48.078319    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6bcbf625-91d3-456a-aa3d-7875407ec889" containerName="config-reloader"
Jul 16 00:42:48.078454 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:48.078324    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6bcbf625-91d3-456a-aa3d-7875407ec889" containerName="config-reloader"
Jul 16 00:42:48.078454 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:48.078330    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6bcbf625-91d3-456a-aa3d-7875407ec889" containerName="kube-rbac-proxy-metric"
Jul 16 00:42:48.078454 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:48.078335    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6bcbf625-91d3-456a-aa3d-7875407ec889" containerName="kube-rbac-proxy-metric"
Jul 16 00:42:48.078454 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:48.078344    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6bcbf625-91d3-456a-aa3d-7875407ec889" containerName="kube-rbac-proxy"
Jul 16 00:42:48.078454 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:48.078350    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6bcbf625-91d3-456a-aa3d-7875407ec889" containerName="kube-rbac-proxy"
Jul 16 00:42:48.078454 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:48.078360    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6bcbf625-91d3-456a-aa3d-7875407ec889" containerName="kube-rbac-proxy-web"
Jul 16 00:42:48.078454 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:48.078368    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6bcbf625-91d3-456a-aa3d-7875407ec889" containerName="kube-rbac-proxy-web"
Jul 16 00:42:48.078454 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:48.078412    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="6bcbf625-91d3-456a-aa3d-7875407ec889" containerName="alertmanager"
Jul 16 00:42:48.078454 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:48.078420    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="6bcbf625-91d3-456a-aa3d-7875407ec889" containerName="kube-rbac-proxy-web"
Jul 16 00:42:48.078454 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:48.078447    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="6bcbf625-91d3-456a-aa3d-7875407ec889" containerName="config-reloader"
Jul 16 00:42:48.078454 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:48.078455    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="6bcbf625-91d3-456a-aa3d-7875407ec889" containerName="kube-rbac-proxy"
Jul 16 00:42:48.078779 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:48.078462    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="6bcbf625-91d3-456a-aa3d-7875407ec889" containerName="kube-rbac-proxy-metric"
Jul 16 00:42:48.078779 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:48.078469    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="6bcbf625-91d3-456a-aa3d-7875407ec889" containerName="prom-label-proxy"
Jul 16 00:42:48.083427 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:48.083410    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 16 00:42:48.085995 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:48.085960    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 16 00:42:48.086138 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:48.086026    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-lqhwj\""
Jul 16 00:42:48.086138 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:48.086042    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 16 00:42:48.086563 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:48.086534    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 16 00:42:48.086672 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:48.086621    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 16 00:42:48.086672 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:48.086659    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 16 00:42:48.086798 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:48.086722    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 16 00:42:48.086980 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:48.086967    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 16 00:42:48.087044 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:48.086988    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 16 00:42:48.092709 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:48.092692    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 16 00:42:48.109615 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:48.109582    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 16 00:42:48.134337 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:48.134311    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/3c93649a-ae28-4bb7-8452-acf2b9b42acd-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"3c93649a-ae28-4bb7-8452-acf2b9b42acd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 00:42:48.134494 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:48.134341    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3c93649a-ae28-4bb7-8452-acf2b9b42acd-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"3c93649a-ae28-4bb7-8452-acf2b9b42acd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 00:42:48.134494 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:48.134360    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3c93649a-ae28-4bb7-8452-acf2b9b42acd-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"3c93649a-ae28-4bb7-8452-acf2b9b42acd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 00:42:48.134494 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:48.134379    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3c93649a-ae28-4bb7-8452-acf2b9b42acd-tls-assets\") pod \"alertmanager-main-0\" (UID: \"3c93649a-ae28-4bb7-8452-acf2b9b42acd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 00:42:48.134494 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:48.134434    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3c93649a-ae28-4bb7-8452-acf2b9b42acd-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"3c93649a-ae28-4bb7-8452-acf2b9b42acd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 00:42:48.134494 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:48.134473    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3c93649a-ae28-4bb7-8452-acf2b9b42acd-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"3c93649a-ae28-4bb7-8452-acf2b9b42acd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 00:42:48.134716 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:48.134497    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c93649a-ae28-4bb7-8452-acf2b9b42acd-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"3c93649a-ae28-4bb7-8452-acf2b9b42acd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 00:42:48.134716 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:48.134526    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flwvt\" (UniqueName: \"kubernetes.io/projected/3c93649a-ae28-4bb7-8452-acf2b9b42acd-kube-api-access-flwvt\") pod \"alertmanager-main-0\" (UID: \"3c93649a-ae28-4bb7-8452-acf2b9b42acd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 00:42:48.134716 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:48.134573    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/3c93649a-ae28-4bb7-8452-acf2b9b42acd-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"3c93649a-ae28-4bb7-8452-acf2b9b42acd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 00:42:48.134716 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:48.134610    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3c93649a-ae28-4bb7-8452-acf2b9b42acd-config-out\") pod \"alertmanager-main-0\" (UID: \"3c93649a-ae28-4bb7-8452-acf2b9b42acd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 00:42:48.134716 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:48.134701    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3c93649a-ae28-4bb7-8452-acf2b9b42acd-config-volume\") pod \"alertmanager-main-0\" (UID: \"3c93649a-ae28-4bb7-8452-acf2b9b42acd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 00:42:48.134897 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:48.134734    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3c93649a-ae28-4bb7-8452-acf2b9b42acd-web-config\") pod \"alertmanager-main-0\" (UID: \"3c93649a-ae28-4bb7-8452-acf2b9b42acd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 00:42:48.134897 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:48.134766    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3c93649a-ae28-4bb7-8452-acf2b9b42acd-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"3c93649a-ae28-4bb7-8452-acf2b9b42acd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 00:42:48.235405 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:48.235375    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3c93649a-ae28-4bb7-8452-acf2b9b42acd-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"3c93649a-ae28-4bb7-8452-acf2b9b42acd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 00:42:48.235572 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:48.235411    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/3c93649a-ae28-4bb7-8452-acf2b9b42acd-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"3c93649a-ae28-4bb7-8452-acf2b9b42acd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 00:42:48.235572 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:48.235436    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3c93649a-ae28-4bb7-8452-acf2b9b42acd-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"3c93649a-ae28-4bb7-8452-acf2b9b42acd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 00:42:48.235572 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:48.235459    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3c93649a-ae28-4bb7-8452-acf2b9b42acd-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"3c93649a-ae28-4bb7-8452-acf2b9b42acd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 00:42:48.235717 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:48.235633    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3c93649a-ae28-4bb7-8452-acf2b9b42acd-tls-assets\") pod \"alertmanager-main-0\" (UID: \"3c93649a-ae28-4bb7-8452-acf2b9b42acd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 00:42:48.235717 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:48.235681    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3c93649a-ae28-4bb7-8452-acf2b9b42acd-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"3c93649a-ae28-4bb7-8452-acf2b9b42acd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 00:42:48.235717 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:48.235712    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3c93649a-ae28-4bb7-8452-acf2b9b42acd-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"3c93649a-ae28-4bb7-8452-acf2b9b42acd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 00:42:48.235891 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:48.235737    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c93649a-ae28-4bb7-8452-acf2b9b42acd-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"3c93649a-ae28-4bb7-8452-acf2b9b42acd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 00:42:48.235891 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:48.235769    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flwvt\" (UniqueName: \"kubernetes.io/projected/3c93649a-ae28-4bb7-8452-acf2b9b42acd-kube-api-access-flwvt\") pod \"alertmanager-main-0\" (UID: \"3c93649a-ae28-4bb7-8452-acf2b9b42acd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 00:42:48.235891 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:48.235822    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/3c93649a-ae28-4bb7-8452-acf2b9b42acd-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"3c93649a-ae28-4bb7-8452-acf2b9b42acd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 00:42:48.236056 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:48.235901    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3c93649a-ae28-4bb7-8452-acf2b9b42acd-config-out\") pod \"alertmanager-main-0\" (UID: \"3c93649a-ae28-4bb7-8452-acf2b9b42acd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 00:42:48.236056 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:48.235960    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3c93649a-ae28-4bb7-8452-acf2b9b42acd-config-volume\") pod \"alertmanager-main-0\" (UID: \"3c93649a-ae28-4bb7-8452-acf2b9b42acd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 00:42:48.236056 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:48.235999    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3c93649a-ae28-4bb7-8452-acf2b9b42acd-web-config\") pod \"alertmanager-main-0\" (UID: \"3c93649a-ae28-4bb7-8452-acf2b9b42acd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 00:42:48.236490 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:48.236449    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3c93649a-ae28-4bb7-8452-acf2b9b42acd-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"3c93649a-ae28-4bb7-8452-acf2b9b42acd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 00:42:48.236789 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:48.236767    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/3c93649a-ae28-4bb7-8452-acf2b9b42acd-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"3c93649a-ae28-4bb7-8452-acf2b9b42acd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 00:42:48.239053 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:48.238435    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3c93649a-ae28-4bb7-8452-acf2b9b42acd-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"3c93649a-ae28-4bb7-8452-acf2b9b42acd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 00:42:48.239053 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:48.238557    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c93649a-ae28-4bb7-8452-acf2b9b42acd-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"3c93649a-ae28-4bb7-8452-acf2b9b42acd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 00:42:48.239053 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:48.238905    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3c93649a-ae28-4bb7-8452-acf2b9b42acd-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"3c93649a-ae28-4bb7-8452-acf2b9b42acd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 00:42:48.239053 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:48.238935    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/3c93649a-ae28-4bb7-8452-acf2b9b42acd-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"3c93649a-ae28-4bb7-8452-acf2b9b42acd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 00:42:48.239053 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:48.238970    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3c93649a-ae28-4bb7-8452-acf2b9b42acd-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"3c93649a-ae28-4bb7-8452-acf2b9b42acd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 00:42:48.239053 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:48.238974    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3c93649a-ae28-4bb7-8452-acf2b9b42acd-web-config\") pod \"alertmanager-main-0\" (UID: \"3c93649a-ae28-4bb7-8452-acf2b9b42acd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 00:42:48.240021 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:48.239999    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3c93649a-ae28-4bb7-8452-acf2b9b42acd-config-out\") pod \"alertmanager-main-0\" (UID: \"3c93649a-ae28-4bb7-8452-acf2b9b42acd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 00:42:48.240100 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:48.240025    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3c93649a-ae28-4bb7-8452-acf2b9b42acd-tls-assets\") pod \"alertmanager-main-0\" (UID: \"3c93649a-ae28-4bb7-8452-acf2b9b42acd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 00:42:48.240453 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:48.240432    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3c93649a-ae28-4bb7-8452-acf2b9b42acd-config-volume\") pod \"alertmanager-main-0\" (UID: \"3c93649a-ae28-4bb7-8452-acf2b9b42acd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 00:42:48.240504 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:48.240440    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3c93649a-ae28-4bb7-8452-acf2b9b42acd-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"3c93649a-ae28-4bb7-8452-acf2b9b42acd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 00:42:48.246847 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:48.246825    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-flwvt\" (UniqueName: \"kubernetes.io/projected/3c93649a-ae28-4bb7-8452-acf2b9b42acd-kube-api-access-flwvt\") pod \"alertmanager-main-0\" (UID: \"3c93649a-ae28-4bb7-8452-acf2b9b42acd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 00:42:48.393301 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:48.393269    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 16 00:42:48.527717 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:48.527670    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 16 00:42:48.529812 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:42:48.529784    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c93649a_ae28_4bb7_8452_acf2b9b42acd.slice/crio-a0f67449a0aa602e6ffd1e2300f53e87971bf40d59daa1bcd436f472efe0957c WatchSource:0}: Error finding container a0f67449a0aa602e6ffd1e2300f53e87971bf40d59daa1bcd436f472efe0957c: Status 404 returned error can't find the container with id a0f67449a0aa602e6ffd1e2300f53e87971bf40d59daa1bcd436f472efe0957c
Jul 16 00:42:48.709349 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:48.709267    2563 generic.go:356] "Generic (PLEG): container finished" podID="3c93649a-ae28-4bb7-8452-acf2b9b42acd" containerID="81f6ec7709ce9210ef91b06acab54640e825424ab1c9da6ec8251a338b1ce5f4" exitCode=0
Jul 16 00:42:48.709349 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:48.709304    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3c93649a-ae28-4bb7-8452-acf2b9b42acd","Type":"ContainerDied","Data":"81f6ec7709ce9210ef91b06acab54640e825424ab1c9da6ec8251a338b1ce5f4"}
Jul 16 00:42:48.709716 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:48.709379    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3c93649a-ae28-4bb7-8452-acf2b9b42acd","Type":"ContainerStarted","Data":"a0f67449a0aa602e6ffd1e2300f53e87971bf40d59daa1bcd436f472efe0957c"}
Jul 16 00:42:49.715822 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:49.715784    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3c93649a-ae28-4bb7-8452-acf2b9b42acd","Type":"ContainerStarted","Data":"69207d5b17fa94815f0a793eed4723adc3e1f381a0ec4fa489ab860763a75540"}
Jul 16 00:42:49.716220 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:49.715852    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3c93649a-ae28-4bb7-8452-acf2b9b42acd","Type":"ContainerStarted","Data":"1a83668fa1cf5cc169a1be744781b3acc5312c3df398b93f764092c1978a34df"}
Jul 16 00:42:49.716220 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:49.715907    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3c93649a-ae28-4bb7-8452-acf2b9b42acd","Type":"ContainerStarted","Data":"3c2d4afcc72e95c01cffb0756461bfc0270d1e9f40126ca6f89906bbb42b5d66"}
Jul 16 00:42:49.716220 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:49.715932    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3c93649a-ae28-4bb7-8452-acf2b9b42acd","Type":"ContainerStarted","Data":"80e3fecbae6ad6ca418d642934987b49b2bbf2f6260c67ff01a529ac2e919a3d"}
Jul 16 00:42:49.716220 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:49.715959    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3c93649a-ae28-4bb7-8452-acf2b9b42acd","Type":"ContainerStarted","Data":"daa67155cab5cc584a10c87e4d4d24422db1d2fcdae656fe70ec9d064f8a5f4d"}
Jul 16 00:42:49.716220 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:49.715981    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3c93649a-ae28-4bb7-8452-acf2b9b42acd","Type":"ContainerStarted","Data":"518e5a8f75144dd3b42d879bffc7956071baa14030b14d7203cec73760825549"}
Jul 16 00:42:49.745911 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:49.745813    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=1.725866146 podStartE2EDuration="1.745793431s" totalImagesPullingTime="0s" totalInitContainerRuntime="19.927285ms" isStatefulPod=false podCreationTimestamp="2026-07-16 00:42:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 00:42:49.744211859 +0000 UTC m=+142.179473600" watchObservedRunningTime="2026-07-16 00:42:49.745793431 +0000 UTC m=+142.181055174"
Jul 16 00:42:50.005667 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:50.005585    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6bcbf625-91d3-456a-aa3d-7875407ec889" path="/var/lib/kubelet/pods/6bcbf625-91d3-456a-aa3d-7875407ec889/volumes"
Jul 16 00:42:50.753420 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:50.753386    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 16 00:42:50.753826 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:50.753691    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="d52ea21c-7d16-4007-a680-b67f3cfbcdc6" containerName="prometheus" containerID="cri-o://96c91fdd8e52ee998ebf81468edc2bb168994118fa5fa82ffe1361c2440d3d1e" gracePeriod=600
Jul 16 00:42:50.753826 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:50.753733    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="d52ea21c-7d16-4007-a680-b67f3cfbcdc6" containerName="kube-rbac-proxy" containerID="cri-o://edfb98ca40b9ca5d4d12e1054bbb357a4bedad235fe36b73fb1b5cdebb18c1e2" gracePeriod=600
Jul 16 00:42:50.753826 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:50.753768    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="d52ea21c-7d16-4007-a680-b67f3cfbcdc6" containerName="thanos-sidecar" containerID="cri-o://da5e0a6e27ccb609f6341ca1d7ae16b2c6b3054dc2a24bfc2145fe9ede975837" gracePeriod=600
Jul 16 00:42:50.753826 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:50.753791    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="d52ea21c-7d16-4007-a680-b67f3cfbcdc6" containerName="config-reloader" containerID="cri-o://3b3d32f5e37e91811bbc20f3c1b425fa39f05f30266673ef09e240b586a20fd7" gracePeriod=600
Jul 16 00:42:50.754096 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:50.753775    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="d52ea21c-7d16-4007-a680-b67f3cfbcdc6" containerName="kube-rbac-proxy-web" containerID="cri-o://73377ca2c9f5f98bdae8ac167ace8f28d9170705feff90bbff87dbf4058659ad" gracePeriod=600
Jul 16 00:42:50.754096 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:50.753769    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="d52ea21c-7d16-4007-a680-b67f3cfbcdc6" containerName="kube-rbac-proxy-thanos" containerID="cri-o://233220bbd44752a0b45a1b01321a7b1d68e89cf5b75590206313f0afa56cc21f" gracePeriod=600
Jul 16 00:42:50.998029 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:50.998004    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 00:42:51.062192 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.062111    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d52ea21c-7d16-4007-a680-b67f3cfbcdc6-web-config\") pod \"d52ea21c-7d16-4007-a680-b67f3cfbcdc6\" (UID: \"d52ea21c-7d16-4007-a680-b67f3cfbcdc6\") "
Jul 16 00:42:51.062192 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.062159    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rdcwd\" (UniqueName: \"kubernetes.io/projected/d52ea21c-7d16-4007-a680-b67f3cfbcdc6-kube-api-access-rdcwd\") pod \"d52ea21c-7d16-4007-a680-b67f3cfbcdc6\" (UID: \"d52ea21c-7d16-4007-a680-b67f3cfbcdc6\") "
Jul 16 00:42:51.062397 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.062230    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d52ea21c-7d16-4007-a680-b67f3cfbcdc6-configmap-serving-certs-ca-bundle\") pod \"d52ea21c-7d16-4007-a680-b67f3cfbcdc6\" (UID: \"d52ea21c-7d16-4007-a680-b67f3cfbcdc6\") "
Jul 16 00:42:51.062397 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.062288    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d52ea21c-7d16-4007-a680-b67f3cfbcdc6-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"d52ea21c-7d16-4007-a680-b67f3cfbcdc6\" (UID: \"d52ea21c-7d16-4007-a680-b67f3cfbcdc6\") "
Jul 16 00:42:51.062397 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.062319    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/d52ea21c-7d16-4007-a680-b67f3cfbcdc6-config\") pod \"d52ea21c-7d16-4007-a680-b67f3cfbcdc6\" (UID: \"d52ea21c-7d16-4007-a680-b67f3cfbcdc6\") "
Jul 16 00:42:51.062397 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.062356    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d52ea21c-7d16-4007-a680-b67f3cfbcdc6-config-out\") pod \"d52ea21c-7d16-4007-a680-b67f3cfbcdc6\" (UID: \"d52ea21c-7d16-4007-a680-b67f3cfbcdc6\") "
Jul 16 00:42:51.062397 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.062386    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d52ea21c-7d16-4007-a680-b67f3cfbcdc6-prometheus-k8s-rulefiles-2\") pod \"d52ea21c-7d16-4007-a680-b67f3cfbcdc6\" (UID: \"d52ea21c-7d16-4007-a680-b67f3cfbcdc6\") "
Jul 16 00:42:51.062640 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.062416    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d52ea21c-7d16-4007-a680-b67f3cfbcdc6-thanos-prometheus-http-client-file\") pod \"d52ea21c-7d16-4007-a680-b67f3cfbcdc6\" (UID: \"d52ea21c-7d16-4007-a680-b67f3cfbcdc6\") "
Jul 16 00:42:51.062640 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.062450    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d52ea21c-7d16-4007-a680-b67f3cfbcdc6-prometheus-k8s-db\") pod \"d52ea21c-7d16-4007-a680-b67f3cfbcdc6\" (UID: \"d52ea21c-7d16-4007-a680-b67f3cfbcdc6\") "
Jul 16 00:42:51.062640 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.062484    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d52ea21c-7d16-4007-a680-b67f3cfbcdc6-prometheus-trusted-ca-bundle\") pod \"d52ea21c-7d16-4007-a680-b67f3cfbcdc6\" (UID: \"d52ea21c-7d16-4007-a680-b67f3cfbcdc6\") "
Jul 16 00:42:51.062640 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.062510    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d52ea21c-7d16-4007-a680-b67f3cfbcdc6-secret-grpc-tls\") pod \"d52ea21c-7d16-4007-a680-b67f3cfbcdc6\" (UID: \"d52ea21c-7d16-4007-a680-b67f3cfbcdc6\") "
Jul 16 00:42:51.062640 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.062534    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d52ea21c-7d16-4007-a680-b67f3cfbcdc6-secret-metrics-client-certs\") pod \"d52ea21c-7d16-4007-a680-b67f3cfbcdc6\" (UID: \"d52ea21c-7d16-4007-a680-b67f3cfbcdc6\") "
Jul 16 00:42:51.062640 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.062556    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d52ea21c-7d16-4007-a680-b67f3cfbcdc6-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "d52ea21c-7d16-4007-a680-b67f3cfbcdc6" (UID: "d52ea21c-7d16-4007-a680-b67f3cfbcdc6"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 00:42:51.062640 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.062627    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d52ea21c-7d16-4007-a680-b67f3cfbcdc6-prometheus-k8s-rulefiles-0\") pod \"d52ea21c-7d16-4007-a680-b67f3cfbcdc6\" (UID: \"d52ea21c-7d16-4007-a680-b67f3cfbcdc6\") "
Jul 16 00:42:51.062986 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.062679    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d52ea21c-7d16-4007-a680-b67f3cfbcdc6-secret-kube-rbac-proxy\") pod \"d52ea21c-7d16-4007-a680-b67f3cfbcdc6\" (UID: \"d52ea21c-7d16-4007-a680-b67f3cfbcdc6\") "
Jul 16 00:42:51.062986 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.062718    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d52ea21c-7d16-4007-a680-b67f3cfbcdc6-configmap-metrics-client-ca\") pod \"d52ea21c-7d16-4007-a680-b67f3cfbcdc6\" (UID: \"d52ea21c-7d16-4007-a680-b67f3cfbcdc6\") "
Jul 16 00:42:51.062986 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.062749    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d52ea21c-7d16-4007-a680-b67f3cfbcdc6-prometheus-k8s-rulefiles-1\") pod \"d52ea21c-7d16-4007-a680-b67f3cfbcdc6\" (UID: \"d52ea21c-7d16-4007-a680-b67f3cfbcdc6\") "
Jul 16 00:42:51.062986 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.062763    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d52ea21c-7d16-4007-a680-b67f3cfbcdc6-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "d52ea21c-7d16-4007-a680-b67f3cfbcdc6" (UID: "d52ea21c-7d16-4007-a680-b67f3cfbcdc6"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 00:42:51.062986 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.062777    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d52ea21c-7d16-4007-a680-b67f3cfbcdc6-tls-assets\") pod \"d52ea21c-7d16-4007-a680-b67f3cfbcdc6\" (UID: \"d52ea21c-7d16-4007-a680-b67f3cfbcdc6\") "
Jul 16 00:42:51.062986 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.062971    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d52ea21c-7d16-4007-a680-b67f3cfbcdc6-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"d52ea21c-7d16-4007-a680-b67f3cfbcdc6\" (UID: \"d52ea21c-7d16-4007-a680-b67f3cfbcdc6\") "
Jul 16 00:42:51.063269 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.063007    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d52ea21c-7d16-4007-a680-b67f3cfbcdc6-secret-prometheus-k8s-tls\") pod \"d52ea21c-7d16-4007-a680-b67f3cfbcdc6\" (UID: \"d52ea21c-7d16-4007-a680-b67f3cfbcdc6\") "
Jul 16 00:42:51.063269 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.063039    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d52ea21c-7d16-4007-a680-b67f3cfbcdc6-configmap-kubelet-serving-ca-bundle\") pod \"d52ea21c-7d16-4007-a680-b67f3cfbcdc6\" (UID: \"d52ea21c-7d16-4007-a680-b67f3cfbcdc6\") "
Jul 16 00:42:51.063364 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.063315    2563 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d52ea21c-7d16-4007-a680-b67f3cfbcdc6-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-143-65.ec2.internal\" DevicePath \"\""
Jul 16 00:42:51.063364 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.063333    2563 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d52ea21c-7d16-4007-a680-b67f3cfbcdc6-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-143-65.ec2.internal\" DevicePath \"\""
Jul 16 00:42:51.067934 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.063637    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d52ea21c-7d16-4007-a680-b67f3cfbcdc6-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "d52ea21c-7d16-4007-a680-b67f3cfbcdc6" (UID: "d52ea21c-7d16-4007-a680-b67f3cfbcdc6"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 00:42:51.067934 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.063892    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d52ea21c-7d16-4007-a680-b67f3cfbcdc6-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "d52ea21c-7d16-4007-a680-b67f3cfbcdc6" (UID: "d52ea21c-7d16-4007-a680-b67f3cfbcdc6"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 00:42:51.067934 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.065787    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d52ea21c-7d16-4007-a680-b67f3cfbcdc6-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "d52ea21c-7d16-4007-a680-b67f3cfbcdc6" (UID: "d52ea21c-7d16-4007-a680-b67f3cfbcdc6"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 00:42:51.067934 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.066080    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d52ea21c-7d16-4007-a680-b67f3cfbcdc6-kube-api-access-rdcwd" (OuterVolumeSpecName: "kube-api-access-rdcwd") pod "d52ea21c-7d16-4007-a680-b67f3cfbcdc6" (UID: "d52ea21c-7d16-4007-a680-b67f3cfbcdc6"). InnerVolumeSpecName "kube-api-access-rdcwd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 00:42:51.067934 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.066480    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d52ea21c-7d16-4007-a680-b67f3cfbcdc6-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "d52ea21c-7d16-4007-a680-b67f3cfbcdc6" (UID: "d52ea21c-7d16-4007-a680-b67f3cfbcdc6"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 00:42:51.067934 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.066645    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d52ea21c-7d16-4007-a680-b67f3cfbcdc6-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "d52ea21c-7d16-4007-a680-b67f3cfbcdc6" (UID: "d52ea21c-7d16-4007-a680-b67f3cfbcdc6"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 00:42:51.067934 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.066666    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d52ea21c-7d16-4007-a680-b67f3cfbcdc6-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "d52ea21c-7d16-4007-a680-b67f3cfbcdc6" (UID: "d52ea21c-7d16-4007-a680-b67f3cfbcdc6"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 00:42:51.067934 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.067080    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d52ea21c-7d16-4007-a680-b67f3cfbcdc6-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "d52ea21c-7d16-4007-a680-b67f3cfbcdc6" (UID: "d52ea21c-7d16-4007-a680-b67f3cfbcdc6"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 00:42:51.067934 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.067094    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d52ea21c-7d16-4007-a680-b67f3cfbcdc6-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "d52ea21c-7d16-4007-a680-b67f3cfbcdc6" (UID: "d52ea21c-7d16-4007-a680-b67f3cfbcdc6"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 00:42:51.067934 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.067188    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d52ea21c-7d16-4007-a680-b67f3cfbcdc6-config" (OuterVolumeSpecName: "config") pod "d52ea21c-7d16-4007-a680-b67f3cfbcdc6" (UID: "d52ea21c-7d16-4007-a680-b67f3cfbcdc6"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 00:42:51.067934 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.067270    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d52ea21c-7d16-4007-a680-b67f3cfbcdc6-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "d52ea21c-7d16-4007-a680-b67f3cfbcdc6" (UID: "d52ea21c-7d16-4007-a680-b67f3cfbcdc6"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 00:42:51.067934 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.067289    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d52ea21c-7d16-4007-a680-b67f3cfbcdc6-config-out" (OuterVolumeSpecName: "config-out") pod "d52ea21c-7d16-4007-a680-b67f3cfbcdc6" (UID: "d52ea21c-7d16-4007-a680-b67f3cfbcdc6"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 00:42:51.067934 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.067418    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d52ea21c-7d16-4007-a680-b67f3cfbcdc6-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "d52ea21c-7d16-4007-a680-b67f3cfbcdc6" (UID: "d52ea21c-7d16-4007-a680-b67f3cfbcdc6"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 00:42:51.068758 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.068510    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d52ea21c-7d16-4007-a680-b67f3cfbcdc6-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "d52ea21c-7d16-4007-a680-b67f3cfbcdc6" (UID: "d52ea21c-7d16-4007-a680-b67f3cfbcdc6"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 00:42:51.068856 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.068835    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d52ea21c-7d16-4007-a680-b67f3cfbcdc6-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "d52ea21c-7d16-4007-a680-b67f3cfbcdc6" (UID: "d52ea21c-7d16-4007-a680-b67f3cfbcdc6"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 00:42:51.068953 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.068927    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d52ea21c-7d16-4007-a680-b67f3cfbcdc6-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "d52ea21c-7d16-4007-a680-b67f3cfbcdc6" (UID: "d52ea21c-7d16-4007-a680-b67f3cfbcdc6"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 00:42:51.069003 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.068985    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d52ea21c-7d16-4007-a680-b67f3cfbcdc6-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "d52ea21c-7d16-4007-a680-b67f3cfbcdc6" (UID: "d52ea21c-7d16-4007-a680-b67f3cfbcdc6"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 00:42:51.076358 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.076331    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d52ea21c-7d16-4007-a680-b67f3cfbcdc6-web-config" (OuterVolumeSpecName: "web-config") pod "d52ea21c-7d16-4007-a680-b67f3cfbcdc6" (UID: "d52ea21c-7d16-4007-a680-b67f3cfbcdc6"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 00:42:51.164492 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.164445    2563 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d52ea21c-7d16-4007-a680-b67f3cfbcdc6-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-143-65.ec2.internal\" DevicePath \"\""
Jul 16 00:42:51.164492 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.164482    2563 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/d52ea21c-7d16-4007-a680-b67f3cfbcdc6-config\") on node \"ip-10-0-143-65.ec2.internal\" DevicePath \"\""
Jul 16 00:42:51.164492 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.164492    2563 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d52ea21c-7d16-4007-a680-b67f3cfbcdc6-config-out\") on node \"ip-10-0-143-65.ec2.internal\" DevicePath \"\""
Jul 16 00:42:51.164492 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.164503    2563 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d52ea21c-7d16-4007-a680-b67f3cfbcdc6-thanos-prometheus-http-client-file\") on node \"ip-10-0-143-65.ec2.internal\" DevicePath \"\""
Jul 16 00:42:51.164741 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.164512    2563 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d52ea21c-7d16-4007-a680-b67f3cfbcdc6-prometheus-k8s-db\") on node \"ip-10-0-143-65.ec2.internal\" DevicePath \"\""
Jul 16 00:42:51.164741 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.164521    2563 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d52ea21c-7d16-4007-a680-b67f3cfbcdc6-prometheus-trusted-ca-bundle\") on node \"ip-10-0-143-65.ec2.internal\" DevicePath \"\""
Jul 16 00:42:51.164741 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.164529    2563 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d52ea21c-7d16-4007-a680-b67f3cfbcdc6-secret-grpc-tls\") on node \"ip-10-0-143-65.ec2.internal\" DevicePath \"\""
Jul 16 00:42:51.164741 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.164537    2563 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d52ea21c-7d16-4007-a680-b67f3cfbcdc6-secret-metrics-client-certs\") on node \"ip-10-0-143-65.ec2.internal\" DevicePath \"\""
Jul 16 00:42:51.164741 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.164545    2563 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d52ea21c-7d16-4007-a680-b67f3cfbcdc6-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-143-65.ec2.internal\" DevicePath \"\""
Jul 16 00:42:51.164741 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.164553    2563 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d52ea21c-7d16-4007-a680-b67f3cfbcdc6-secret-kube-rbac-proxy\") on node \"ip-10-0-143-65.ec2.internal\" DevicePath \"\""
Jul 16 00:42:51.164741 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.164561    2563 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d52ea21c-7d16-4007-a680-b67f3cfbcdc6-configmap-metrics-client-ca\") on node \"ip-10-0-143-65.ec2.internal\" DevicePath \"\""
Jul 16 00:42:51.164741 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.164570    2563 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d52ea21c-7d16-4007-a680-b67f3cfbcdc6-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-143-65.ec2.internal\" DevicePath \"\""
Jul 16 00:42:51.164741 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.164579    2563 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d52ea21c-7d16-4007-a680-b67f3cfbcdc6-tls-assets\") on node \"ip-10-0-143-65.ec2.internal\" DevicePath \"\""
Jul 16 00:42:51.164741 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.164587    2563 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d52ea21c-7d16-4007-a680-b67f3cfbcdc6-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-143-65.ec2.internal\" DevicePath \"\""
Jul 16 00:42:51.164741 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.164595    2563 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d52ea21c-7d16-4007-a680-b67f3cfbcdc6-secret-prometheus-k8s-tls\") on node \"ip-10-0-143-65.ec2.internal\" DevicePath \"\""
Jul 16 00:42:51.164741 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.164605    2563 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d52ea21c-7d16-4007-a680-b67f3cfbcdc6-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-143-65.ec2.internal\" DevicePath \"\""
Jul 16 00:42:51.164741 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.164613    2563 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d52ea21c-7d16-4007-a680-b67f3cfbcdc6-web-config\") on node \"ip-10-0-143-65.ec2.internal\" DevicePath \"\""
Jul 16 00:42:51.164741 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.164621    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdcwd\" (UniqueName: \"kubernetes.io/projected/d52ea21c-7d16-4007-a680-b67f3cfbcdc6-kube-api-access-rdcwd\") on node \"ip-10-0-143-65.ec2.internal\" DevicePath \"\""
Jul 16 00:42:51.726823 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.726785    2563 generic.go:356] "Generic (PLEG): container finished" podID="d52ea21c-7d16-4007-a680-b67f3cfbcdc6" containerID="233220bbd44752a0b45a1b01321a7b1d68e89cf5b75590206313f0afa56cc21f" exitCode=0
Jul 16 00:42:51.726823 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.726811    2563 generic.go:356] "Generic (PLEG): container finished" podID="d52ea21c-7d16-4007-a680-b67f3cfbcdc6" containerID="edfb98ca40b9ca5d4d12e1054bbb357a4bedad235fe36b73fb1b5cdebb18c1e2" exitCode=0
Jul 16 00:42:51.726823 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.726817    2563 generic.go:356] "Generic (PLEG): container finished" podID="d52ea21c-7d16-4007-a680-b67f3cfbcdc6" containerID="73377ca2c9f5f98bdae8ac167ace8f28d9170705feff90bbff87dbf4058659ad" exitCode=0
Jul 16 00:42:51.726823 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.726825    2563 generic.go:356] "Generic (PLEG): container finished" podID="d52ea21c-7d16-4007-a680-b67f3cfbcdc6" containerID="da5e0a6e27ccb609f6341ca1d7ae16b2c6b3054dc2a24bfc2145fe9ede975837" exitCode=0
Jul 16 00:42:51.726823 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.726832    2563 generic.go:356] "Generic (PLEG): container finished" podID="d52ea21c-7d16-4007-a680-b67f3cfbcdc6" containerID="3b3d32f5e37e91811bbc20f3c1b425fa39f05f30266673ef09e240b586a20fd7" exitCode=0
Jul 16 00:42:51.726823 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.726837    2563 generic.go:356] "Generic (PLEG): container finished" podID="d52ea21c-7d16-4007-a680-b67f3cfbcdc6" containerID="96c91fdd8e52ee998ebf81468edc2bb168994118fa5fa82ffe1361c2440d3d1e" exitCode=0
Jul 16 00:42:51.727220 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.726892    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d52ea21c-7d16-4007-a680-b67f3cfbcdc6","Type":"ContainerDied","Data":"233220bbd44752a0b45a1b01321a7b1d68e89cf5b75590206313f0afa56cc21f"}
Jul 16 00:42:51.727220 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.726903    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 00:42:51.727220 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.726990    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d52ea21c-7d16-4007-a680-b67f3cfbcdc6","Type":"ContainerDied","Data":"edfb98ca40b9ca5d4d12e1054bbb357a4bedad235fe36b73fb1b5cdebb18c1e2"}
Jul 16 00:42:51.727220 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.727018    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d52ea21c-7d16-4007-a680-b67f3cfbcdc6","Type":"ContainerDied","Data":"73377ca2c9f5f98bdae8ac167ace8f28d9170705feff90bbff87dbf4058659ad"}
Jul 16 00:42:51.727220 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.727045    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d52ea21c-7d16-4007-a680-b67f3cfbcdc6","Type":"ContainerDied","Data":"da5e0a6e27ccb609f6341ca1d7ae16b2c6b3054dc2a24bfc2145fe9ede975837"}
Jul 16 00:42:51.727220 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.727061    2563 scope.go:117] "RemoveContainer" containerID="233220bbd44752a0b45a1b01321a7b1d68e89cf5b75590206313f0afa56cc21f"
Jul 16 00:42:51.727220 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.727069    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d52ea21c-7d16-4007-a680-b67f3cfbcdc6","Type":"ContainerDied","Data":"3b3d32f5e37e91811bbc20f3c1b425fa39f05f30266673ef09e240b586a20fd7"}
Jul 16 00:42:51.727220 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.727095    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d52ea21c-7d16-4007-a680-b67f3cfbcdc6","Type":"ContainerDied","Data":"96c91fdd8e52ee998ebf81468edc2bb168994118fa5fa82ffe1361c2440d3d1e"}
Jul 16 00:42:51.727220 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.727118    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d52ea21c-7d16-4007-a680-b67f3cfbcdc6","Type":"ContainerDied","Data":"5db6ddf319439353abd512a2b30f96c96df3fe238004825352ac2409e0d2912b"}
Jul 16 00:42:51.734390 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.734374    2563 scope.go:117] "RemoveContainer" containerID="edfb98ca40b9ca5d4d12e1054bbb357a4bedad235fe36b73fb1b5cdebb18c1e2"
Jul 16 00:42:51.741437 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.741418    2563 scope.go:117] "RemoveContainer" containerID="73377ca2c9f5f98bdae8ac167ace8f28d9170705feff90bbff87dbf4058659ad"
Jul 16 00:42:51.747968 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.747948    2563 scope.go:117] "RemoveContainer" containerID="da5e0a6e27ccb609f6341ca1d7ae16b2c6b3054dc2a24bfc2145fe9ede975837"
Jul 16 00:42:51.752638 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.752618    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 16 00:42:51.755534 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.755404    2563 scope.go:117] "RemoveContainer" containerID="3b3d32f5e37e91811bbc20f3c1b425fa39f05f30266673ef09e240b586a20fd7"
Jul 16 00:42:51.757088 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.757066    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 16 00:42:51.762245 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.762224    2563 scope.go:117] "RemoveContainer" containerID="96c91fdd8e52ee998ebf81468edc2bb168994118fa5fa82ffe1361c2440d3d1e"
Jul 16 00:42:51.780917 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.780896    2563 scope.go:117] "RemoveContainer" containerID="602de56208d0126d303fe4020a5be610e107e35d39c452204f4aab86032c3710"
Jul 16 00:42:51.787442 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.787412    2563 scope.go:117] "RemoveContainer" containerID="233220bbd44752a0b45a1b01321a7b1d68e89cf5b75590206313f0afa56cc21f"
Jul 16 00:42:51.787697 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:42:51.787679    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"233220bbd44752a0b45a1b01321a7b1d68e89cf5b75590206313f0afa56cc21f\": container with ID starting with 233220bbd44752a0b45a1b01321a7b1d68e89cf5b75590206313f0afa56cc21f not found: ID does not exist" containerID="233220bbd44752a0b45a1b01321a7b1d68e89cf5b75590206313f0afa56cc21f"
Jul 16 00:42:51.787758 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.787706    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"233220bbd44752a0b45a1b01321a7b1d68e89cf5b75590206313f0afa56cc21f"} err="failed to get container status \"233220bbd44752a0b45a1b01321a7b1d68e89cf5b75590206313f0afa56cc21f\": rpc error: code = NotFound desc = could not find container \"233220bbd44752a0b45a1b01321a7b1d68e89cf5b75590206313f0afa56cc21f\": container with ID starting with 233220bbd44752a0b45a1b01321a7b1d68e89cf5b75590206313f0afa56cc21f not found: ID does not exist"
Jul 16 00:42:51.787758 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.787723    2563 scope.go:117] "RemoveContainer" containerID="edfb98ca40b9ca5d4d12e1054bbb357a4bedad235fe36b73fb1b5cdebb18c1e2"
Jul 16 00:42:51.787982 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:42:51.787957    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edfb98ca40b9ca5d4d12e1054bbb357a4bedad235fe36b73fb1b5cdebb18c1e2\": container with ID starting with edfb98ca40b9ca5d4d12e1054bbb357a4bedad235fe36b73fb1b5cdebb18c1e2 not found: ID does not exist" containerID="edfb98ca40b9ca5d4d12e1054bbb357a4bedad235fe36b73fb1b5cdebb18c1e2"
Jul 16 00:42:51.788103 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.787989    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edfb98ca40b9ca5d4d12e1054bbb357a4bedad235fe36b73fb1b5cdebb18c1e2"} err="failed to get container status \"edfb98ca40b9ca5d4d12e1054bbb357a4bedad235fe36b73fb1b5cdebb18c1e2\": rpc error: code = NotFound desc = could not find container \"edfb98ca40b9ca5d4d12e1054bbb357a4bedad235fe36b73fb1b5cdebb18c1e2\": container with ID starting with edfb98ca40b9ca5d4d12e1054bbb357a4bedad235fe36b73fb1b5cdebb18c1e2 not found: ID does not exist"
Jul 16 00:42:51.788103 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.788011    2563 scope.go:117] "RemoveContainer" containerID="73377ca2c9f5f98bdae8ac167ace8f28d9170705feff90bbff87dbf4058659ad"
Jul 16 00:42:51.788259 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:42:51.788244    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73377ca2c9f5f98bdae8ac167ace8f28d9170705feff90bbff87dbf4058659ad\": container with ID starting with 73377ca2c9f5f98bdae8ac167ace8f28d9170705feff90bbff87dbf4058659ad not found: ID does not exist" containerID="73377ca2c9f5f98bdae8ac167ace8f28d9170705feff90bbff87dbf4058659ad"
Jul 16 00:42:51.788298 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.788262    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73377ca2c9f5f98bdae8ac167ace8f28d9170705feff90bbff87dbf4058659ad"} err="failed to get container status \"73377ca2c9f5f98bdae8ac167ace8f28d9170705feff90bbff87dbf4058659ad\": rpc error: code = NotFound desc = could not find container \"73377ca2c9f5f98bdae8ac167ace8f28d9170705feff90bbff87dbf4058659ad\": container with ID starting with 73377ca2c9f5f98bdae8ac167ace8f28d9170705feff90bbff87dbf4058659ad not found: ID does not exist"
Jul 16 00:42:51.788298 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.788276    2563 scope.go:117] "RemoveContainer" containerID="da5e0a6e27ccb609f6341ca1d7ae16b2c6b3054dc2a24bfc2145fe9ede975837"
Jul 16 00:42:51.788478 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:42:51.788463    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da5e0a6e27ccb609f6341ca1d7ae16b2c6b3054dc2a24bfc2145fe9ede975837\": container with ID starting with da5e0a6e27ccb609f6341ca1d7ae16b2c6b3054dc2a24bfc2145fe9ede975837 not found: ID does not exist" containerID="da5e0a6e27ccb609f6341ca1d7ae16b2c6b3054dc2a24bfc2145fe9ede975837"
Jul 16 00:42:51.788531 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.788481    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da5e0a6e27ccb609f6341ca1d7ae16b2c6b3054dc2a24bfc2145fe9ede975837"} err="failed to get container status \"da5e0a6e27ccb609f6341ca1d7ae16b2c6b3054dc2a24bfc2145fe9ede975837\": rpc error: code = NotFound desc = could not find container \"da5e0a6e27ccb609f6341ca1d7ae16b2c6b3054dc2a24bfc2145fe9ede975837\": container with ID starting with da5e0a6e27ccb609f6341ca1d7ae16b2c6b3054dc2a24bfc2145fe9ede975837 not found: ID does not exist"
Jul 16 00:42:51.788531 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.788499    2563 scope.go:117] "RemoveContainer" containerID="3b3d32f5e37e91811bbc20f3c1b425fa39f05f30266673ef09e240b586a20fd7"
Jul 16 00:42:51.788691 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:42:51.788671    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b3d32f5e37e91811bbc20f3c1b425fa39f05f30266673ef09e240b586a20fd7\": container with ID starting with 3b3d32f5e37e91811bbc20f3c1b425fa39f05f30266673ef09e240b586a20fd7 not found: ID does not exist" containerID="3b3d32f5e37e91811bbc20f3c1b425fa39f05f30266673ef09e240b586a20fd7"
Jul 16 00:42:51.788734 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.788698    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b3d32f5e37e91811bbc20f3c1b425fa39f05f30266673ef09e240b586a20fd7"} err="failed to get container status \"3b3d32f5e37e91811bbc20f3c1b425fa39f05f30266673ef09e240b586a20fd7\": rpc error: code = NotFound desc = could not find container \"3b3d32f5e37e91811bbc20f3c1b425fa39f05f30266673ef09e240b586a20fd7\": container with ID starting with 3b3d32f5e37e91811bbc20f3c1b425fa39f05f30266673ef09e240b586a20fd7 not found: ID does not exist"
Jul 16 00:42:51.788734 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.788712    2563 scope.go:117] "RemoveContainer" containerID="96c91fdd8e52ee998ebf81468edc2bb168994118fa5fa82ffe1361c2440d3d1e"
Jul 16 00:42:51.788991 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:42:51.788976    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96c91fdd8e52ee998ebf81468edc2bb168994118fa5fa82ffe1361c2440d3d1e\": container with ID starting with 96c91fdd8e52ee998ebf81468edc2bb168994118fa5fa82ffe1361c2440d3d1e not found: ID does not exist" containerID="96c91fdd8e52ee998ebf81468edc2bb168994118fa5fa82ffe1361c2440d3d1e"
Jul 16 00:42:51.789053 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.788994    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96c91fdd8e52ee998ebf81468edc2bb168994118fa5fa82ffe1361c2440d3d1e"} err="failed to get container status \"96c91fdd8e52ee998ebf81468edc2bb168994118fa5fa82ffe1361c2440d3d1e\": rpc error: code = NotFound desc = could not find container \"96c91fdd8e52ee998ebf81468edc2bb168994118fa5fa82ffe1361c2440d3d1e\": container with ID starting with 96c91fdd8e52ee998ebf81468edc2bb168994118fa5fa82ffe1361c2440d3d1e not found: ID does not exist"
Jul 16 00:42:51.789053 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.789009    2563 scope.go:117] "RemoveContainer" containerID="602de56208d0126d303fe4020a5be610e107e35d39c452204f4aab86032c3710"
Jul 16 00:42:51.789213 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:42:51.789197    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"602de56208d0126d303fe4020a5be610e107e35d39c452204f4aab86032c3710\": container with ID starting with 602de56208d0126d303fe4020a5be610e107e35d39c452204f4aab86032c3710 not found: ID does not exist" containerID="602de56208d0126d303fe4020a5be610e107e35d39c452204f4aab86032c3710"
Jul 16 00:42:51.789253 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.789216    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"602de56208d0126d303fe4020a5be610e107e35d39c452204f4aab86032c3710"} err="failed to get container status \"602de56208d0126d303fe4020a5be610e107e35d39c452204f4aab86032c3710\": rpc error: code = NotFound desc = could not find container \"602de56208d0126d303fe4020a5be610e107e35d39c452204f4aab86032c3710\": container with ID starting with 602de56208d0126d303fe4020a5be610e107e35d39c452204f4aab86032c3710 not found: ID does not exist"
Jul 16 00:42:51.789253 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.789228    2563 scope.go:117] "RemoveContainer" containerID="233220bbd44752a0b45a1b01321a7b1d68e89cf5b75590206313f0afa56cc21f"
Jul 16 00:42:51.789460 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.789430    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"233220bbd44752a0b45a1b01321a7b1d68e89cf5b75590206313f0afa56cc21f"} err="failed to get container status \"233220bbd44752a0b45a1b01321a7b1d68e89cf5b75590206313f0afa56cc21f\": rpc error: code = NotFound desc = could not find container \"233220bbd44752a0b45a1b01321a7b1d68e89cf5b75590206313f0afa56cc21f\": container with ID starting with 233220bbd44752a0b45a1b01321a7b1d68e89cf5b75590206313f0afa56cc21f not found: ID does not exist"
Jul 16 00:42:51.789460 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.789445    2563 scope.go:117] "RemoveContainer" containerID="edfb98ca40b9ca5d4d12e1054bbb357a4bedad235fe36b73fb1b5cdebb18c1e2"
Jul 16 00:42:51.789637 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.789622    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edfb98ca40b9ca5d4d12e1054bbb357a4bedad235fe36b73fb1b5cdebb18c1e2"} err="failed to get container status \"edfb98ca40b9ca5d4d12e1054bbb357a4bedad235fe36b73fb1b5cdebb18c1e2\": rpc error: code = NotFound desc = could not find container \"edfb98ca40b9ca5d4d12e1054bbb357a4bedad235fe36b73fb1b5cdebb18c1e2\": container with ID starting with edfb98ca40b9ca5d4d12e1054bbb357a4bedad235fe36b73fb1b5cdebb18c1e2 not found: ID does not exist"
Jul 16 00:42:51.789637 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.789636    2563 scope.go:117] "RemoveContainer" containerID="73377ca2c9f5f98bdae8ac167ace8f28d9170705feff90bbff87dbf4058659ad"
Jul 16 00:42:51.789786 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.789772    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73377ca2c9f5f98bdae8ac167ace8f28d9170705feff90bbff87dbf4058659ad"} err="failed to get container status \"73377ca2c9f5f98bdae8ac167ace8f28d9170705feff90bbff87dbf4058659ad\": rpc error: code = NotFound desc = could not find container \"73377ca2c9f5f98bdae8ac167ace8f28d9170705feff90bbff87dbf4058659ad\": container with ID starting with 73377ca2c9f5f98bdae8ac167ace8f28d9170705feff90bbff87dbf4058659ad not found: ID does not exist"
Jul 16 00:42:51.789841 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.789794    2563 scope.go:117] "RemoveContainer" containerID="da5e0a6e27ccb609f6341ca1d7ae16b2c6b3054dc2a24bfc2145fe9ede975837"
Jul 16 00:42:51.790025 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.790005    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da5e0a6e27ccb609f6341ca1d7ae16b2c6b3054dc2a24bfc2145fe9ede975837"} err="failed to get container status \"da5e0a6e27ccb609f6341ca1d7ae16b2c6b3054dc2a24bfc2145fe9ede975837\": rpc error: code = NotFound desc = could not find container \"da5e0a6e27ccb609f6341ca1d7ae16b2c6b3054dc2a24bfc2145fe9ede975837\": container with ID starting with da5e0a6e27ccb609f6341ca1d7ae16b2c6b3054dc2a24bfc2145fe9ede975837 not found: ID does not exist"
Jul 16 00:42:51.790079 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.790028    2563 scope.go:117] "RemoveContainer" containerID="3b3d32f5e37e91811bbc20f3c1b425fa39f05f30266673ef09e240b586a20fd7"
Jul 16 00:42:51.790232 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.790217    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b3d32f5e37e91811bbc20f3c1b425fa39f05f30266673ef09e240b586a20fd7"} err="failed to get container status \"3b3d32f5e37e91811bbc20f3c1b425fa39f05f30266673ef09e240b586a20fd7\": rpc error: code = NotFound desc = could not find container \"3b3d32f5e37e91811bbc20f3c1b425fa39f05f30266673ef09e240b586a20fd7\": container with ID starting with 3b3d32f5e37e91811bbc20f3c1b425fa39f05f30266673ef09e240b586a20fd7 not found: ID does not exist"
Jul 16 00:42:51.790285 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.790232    2563 scope.go:117] "RemoveContainer" containerID="96c91fdd8e52ee998ebf81468edc2bb168994118fa5fa82ffe1361c2440d3d1e"
Jul 16 00:42:51.790427 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.790410    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96c91fdd8e52ee998ebf81468edc2bb168994118fa5fa82ffe1361c2440d3d1e"} err="failed to get container status \"96c91fdd8e52ee998ebf81468edc2bb168994118fa5fa82ffe1361c2440d3d1e\": rpc error: code = NotFound desc = could not find container \"96c91fdd8e52ee998ebf81468edc2bb168994118fa5fa82ffe1361c2440d3d1e\": container with ID starting with 96c91fdd8e52ee998ebf81468edc2bb168994118fa5fa82ffe1361c2440d3d1e not found: ID does not exist"
Jul 16 00:42:51.790469 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.790428    2563 scope.go:117] "RemoveContainer" containerID="602de56208d0126d303fe4020a5be610e107e35d39c452204f4aab86032c3710"
Jul 16 00:42:51.790619 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.790603    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"602de56208d0126d303fe4020a5be610e107e35d39c452204f4aab86032c3710"} err="failed to get container status \"602de56208d0126d303fe4020a5be610e107e35d39c452204f4aab86032c3710\": rpc error: code = NotFound desc = could not find container \"602de56208d0126d303fe4020a5be610e107e35d39c452204f4aab86032c3710\": container with ID starting with 602de56208d0126d303fe4020a5be610e107e35d39c452204f4aab86032c3710 not found: ID does not exist"
Jul 16 00:42:51.790619 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.790618    2563 scope.go:117] "RemoveContainer" containerID="233220bbd44752a0b45a1b01321a7b1d68e89cf5b75590206313f0afa56cc21f"
Jul 16 00:42:51.790840 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.790823    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"233220bbd44752a0b45a1b01321a7b1d68e89cf5b75590206313f0afa56cc21f"} err="failed to get container status \"233220bbd44752a0b45a1b01321a7b1d68e89cf5b75590206313f0afa56cc21f\": rpc error: code = NotFound desc = could not find container \"233220bbd44752a0b45a1b01321a7b1d68e89cf5b75590206313f0afa56cc21f\": container with ID starting with 233220bbd44752a0b45a1b01321a7b1d68e89cf5b75590206313f0afa56cc21f not found: ID does not exist"
Jul 16 00:42:51.790919 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.790840    2563 scope.go:117] "RemoveContainer" containerID="edfb98ca40b9ca5d4d12e1054bbb357a4bedad235fe36b73fb1b5cdebb18c1e2"
Jul 16 00:42:51.791149 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.791129    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edfb98ca40b9ca5d4d12e1054bbb357a4bedad235fe36b73fb1b5cdebb18c1e2"} err="failed to get container status \"edfb98ca40b9ca5d4d12e1054bbb357a4bedad235fe36b73fb1b5cdebb18c1e2\": rpc error: code = NotFound desc = could not find container \"edfb98ca40b9ca5d4d12e1054bbb357a4bedad235fe36b73fb1b5cdebb18c1e2\": container with ID starting with edfb98ca40b9ca5d4d12e1054bbb357a4bedad235fe36b73fb1b5cdebb18c1e2 not found: ID does not exist"
Jul 16 00:42:51.791205 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.791150    2563 scope.go:117] "RemoveContainer" containerID="73377ca2c9f5f98bdae8ac167ace8f28d9170705feff90bbff87dbf4058659ad"
Jul 16 00:42:51.791361 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.791345    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73377ca2c9f5f98bdae8ac167ace8f28d9170705feff90bbff87dbf4058659ad"} err="failed to get container status \"73377ca2c9f5f98bdae8ac167ace8f28d9170705feff90bbff87dbf4058659ad\": rpc error: code = NotFound desc = could not find container \"73377ca2c9f5f98bdae8ac167ace8f28d9170705feff90bbff87dbf4058659ad\": container with ID starting with 73377ca2c9f5f98bdae8ac167ace8f28d9170705feff90bbff87dbf4058659ad not found: ID does not exist"
Jul 16 00:42:51.791403 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.791362    2563 scope.go:117] "RemoveContainer" containerID="da5e0a6e27ccb609f6341ca1d7ae16b2c6b3054dc2a24bfc2145fe9ede975837"
Jul 16 00:42:51.791555 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.791539    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da5e0a6e27ccb609f6341ca1d7ae16b2c6b3054dc2a24bfc2145fe9ede975837"} err="failed to get container status \"da5e0a6e27ccb609f6341ca1d7ae16b2c6b3054dc2a24bfc2145fe9ede975837\": rpc error: code = NotFound desc = could not find container \"da5e0a6e27ccb609f6341ca1d7ae16b2c6b3054dc2a24bfc2145fe9ede975837\": container with ID starting with da5e0a6e27ccb609f6341ca1d7ae16b2c6b3054dc2a24bfc2145fe9ede975837 not found: ID does not exist"
Jul 16 00:42:51.791555 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.791553    2563 scope.go:117] "RemoveContainer" containerID="3b3d32f5e37e91811bbc20f3c1b425fa39f05f30266673ef09e240b586a20fd7"
Jul 16 00:42:51.791739 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.791722    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b3d32f5e37e91811bbc20f3c1b425fa39f05f30266673ef09e240b586a20fd7"} err="failed to get container status \"3b3d32f5e37e91811bbc20f3c1b425fa39f05f30266673ef09e240b586a20fd7\": rpc error: code = NotFound desc = could not find container \"3b3d32f5e37e91811bbc20f3c1b425fa39f05f30266673ef09e240b586a20fd7\": container with ID starting with 3b3d32f5e37e91811bbc20f3c1b425fa39f05f30266673ef09e240b586a20fd7 not found: ID does not exist"
Jul 16 00:42:51.791783 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.791739    2563 scope.go:117] "RemoveContainer" containerID="96c91fdd8e52ee998ebf81468edc2bb168994118fa5fa82ffe1361c2440d3d1e"
Jul 16 00:42:51.791937 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.791917    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96c91fdd8e52ee998ebf81468edc2bb168994118fa5fa82ffe1361c2440d3d1e"} err="failed to get container status \"96c91fdd8e52ee998ebf81468edc2bb168994118fa5fa82ffe1361c2440d3d1e\": rpc error: code = NotFound desc = could not find container \"96c91fdd8e52ee998ebf81468edc2bb168994118fa5fa82ffe1361c2440d3d1e\": container with ID starting with 96c91fdd8e52ee998ebf81468edc2bb168994118fa5fa82ffe1361c2440d3d1e not found: ID does not exist"
Jul 16 00:42:51.791990 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.791937    2563 scope.go:117] "RemoveContainer" containerID="602de56208d0126d303fe4020a5be610e107e35d39c452204f4aab86032c3710"
Jul 16 00:42:51.792175 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.792155    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"602de56208d0126d303fe4020a5be610e107e35d39c452204f4aab86032c3710"} err="failed to get container status \"602de56208d0126d303fe4020a5be610e107e35d39c452204f4aab86032c3710\": rpc error: code = NotFound desc = could not find container \"602de56208d0126d303fe4020a5be610e107e35d39c452204f4aab86032c3710\": container with ID starting with 602de56208d0126d303fe4020a5be610e107e35d39c452204f4aab86032c3710 not found: ID does not exist"
Jul 16 00:42:51.792247 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.792178    2563 scope.go:117] "RemoveContainer" containerID="233220bbd44752a0b45a1b01321a7b1d68e89cf5b75590206313f0afa56cc21f"
Jul 16 00:42:51.792400 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.792376    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"233220bbd44752a0b45a1b01321a7b1d68e89cf5b75590206313f0afa56cc21f"} err="failed to get container status \"233220bbd44752a0b45a1b01321a7b1d68e89cf5b75590206313f0afa56cc21f\": rpc error: code = NotFound desc = could not find container \"233220bbd44752a0b45a1b01321a7b1d68e89cf5b75590206313f0afa56cc21f\": container with ID starting with 233220bbd44752a0b45a1b01321a7b1d68e89cf5b75590206313f0afa56cc21f not found: ID does not exist"
Jul 16 00:42:51.792400 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.792402    2563 scope.go:117] "RemoveContainer" containerID="edfb98ca40b9ca5d4d12e1054bbb357a4bedad235fe36b73fb1b5cdebb18c1e2"
Jul 16 00:42:51.792617 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.792602    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edfb98ca40b9ca5d4d12e1054bbb357a4bedad235fe36b73fb1b5cdebb18c1e2"} err="failed to get container status \"edfb98ca40b9ca5d4d12e1054bbb357a4bedad235fe36b73fb1b5cdebb18c1e2\": rpc error: code = NotFound desc = could not find container \"edfb98ca40b9ca5d4d12e1054bbb357a4bedad235fe36b73fb1b5cdebb18c1e2\": container with ID starting with edfb98ca40b9ca5d4d12e1054bbb357a4bedad235fe36b73fb1b5cdebb18c1e2 not found: ID does not exist"
Jul 16 00:42:51.792663 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.792617    2563 scope.go:117] "RemoveContainer" containerID="73377ca2c9f5f98bdae8ac167ace8f28d9170705feff90bbff87dbf4058659ad"
Jul 16 00:42:51.792875 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.792842    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73377ca2c9f5f98bdae8ac167ace8f28d9170705feff90bbff87dbf4058659ad"} err="failed to get container status \"73377ca2c9f5f98bdae8ac167ace8f28d9170705feff90bbff87dbf4058659ad\": rpc error: code = NotFound desc = could not find container \"73377ca2c9f5f98bdae8ac167ace8f28d9170705feff90bbff87dbf4058659ad\": container with ID starting with 73377ca2c9f5f98bdae8ac167ace8f28d9170705feff90bbff87dbf4058659ad not found: ID does not exist"
Jul 16 00:42:51.792875 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.792874    2563 scope.go:117] "RemoveContainer" containerID="da5e0a6e27ccb609f6341ca1d7ae16b2c6b3054dc2a24bfc2145fe9ede975837"
Jul 16 00:42:51.793098 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.793081    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da5e0a6e27ccb609f6341ca1d7ae16b2c6b3054dc2a24bfc2145fe9ede975837"} err="failed to get container status \"da5e0a6e27ccb609f6341ca1d7ae16b2c6b3054dc2a24bfc2145fe9ede975837\": rpc error: code = NotFound desc = could not find container \"da5e0a6e27ccb609f6341ca1d7ae16b2c6b3054dc2a24bfc2145fe9ede975837\": container with ID starting with da5e0a6e27ccb609f6341ca1d7ae16b2c6b3054dc2a24bfc2145fe9ede975837 not found: ID does not exist"
Jul 16 00:42:51.793163 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.793099    2563 scope.go:117] "RemoveContainer" containerID="3b3d32f5e37e91811bbc20f3c1b425fa39f05f30266673ef09e240b586a20fd7"
Jul 16 00:42:51.793318 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.793299    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b3d32f5e37e91811bbc20f3c1b425fa39f05f30266673ef09e240b586a20fd7"} err="failed to get container status \"3b3d32f5e37e91811bbc20f3c1b425fa39f05f30266673ef09e240b586a20fd7\": rpc error: code = NotFound desc = could not find container \"3b3d32f5e37e91811bbc20f3c1b425fa39f05f30266673ef09e240b586a20fd7\": container with ID starting with 3b3d32f5e37e91811bbc20f3c1b425fa39f05f30266673ef09e240b586a20fd7 not found: ID does not exist"
Jul 16 00:42:51.793318 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.793314    2563 scope.go:117] "RemoveContainer" containerID="96c91fdd8e52ee998ebf81468edc2bb168994118fa5fa82ffe1361c2440d3d1e"
Jul 16 00:42:51.793517 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.793502    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96c91fdd8e52ee998ebf81468edc2bb168994118fa5fa82ffe1361c2440d3d1e"} err="failed to get container status \"96c91fdd8e52ee998ebf81468edc2bb168994118fa5fa82ffe1361c2440d3d1e\": rpc error: code = NotFound desc = could not find container \"96c91fdd8e52ee998ebf81468edc2bb168994118fa5fa82ffe1361c2440d3d1e\": container with ID starting with 96c91fdd8e52ee998ebf81468edc2bb168994118fa5fa82ffe1361c2440d3d1e not found: ID does not exist"
Jul 16 00:42:51.793561 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.793518    2563 scope.go:117] "RemoveContainer" containerID="602de56208d0126d303fe4020a5be610e107e35d39c452204f4aab86032c3710"
Jul 16 00:42:51.793748 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.793722    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"602de56208d0126d303fe4020a5be610e107e35d39c452204f4aab86032c3710"} err="failed to get container status \"602de56208d0126d303fe4020a5be610e107e35d39c452204f4aab86032c3710\": rpc error: code = NotFound desc = could not find container \"602de56208d0126d303fe4020a5be610e107e35d39c452204f4aab86032c3710\": container with ID starting with 602de56208d0126d303fe4020a5be610e107e35d39c452204f4aab86032c3710 not found: ID does not exist"
Jul 16 00:42:51.793748 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.793748    2563 scope.go:117] "RemoveContainer" containerID="233220bbd44752a0b45a1b01321a7b1d68e89cf5b75590206313f0afa56cc21f"
Jul 16 00:42:51.794000 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.793972    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"233220bbd44752a0b45a1b01321a7b1d68e89cf5b75590206313f0afa56cc21f"} err="failed to get container status \"233220bbd44752a0b45a1b01321a7b1d68e89cf5b75590206313f0afa56cc21f\": rpc error: code = NotFound desc = could not find container \"233220bbd44752a0b45a1b01321a7b1d68e89cf5b75590206313f0afa56cc21f\": container with ID starting with 233220bbd44752a0b45a1b01321a7b1d68e89cf5b75590206313f0afa56cc21f not found: ID does not exist"
Jul 16 00:42:51.794182 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.794003    2563 scope.go:117] "RemoveContainer" containerID="edfb98ca40b9ca5d4d12e1054bbb357a4bedad235fe36b73fb1b5cdebb18c1e2"
Jul 16 00:42:51.794252 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.794219    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edfb98ca40b9ca5d4d12e1054bbb357a4bedad235fe36b73fb1b5cdebb18c1e2"} err="failed to get container status \"edfb98ca40b9ca5d4d12e1054bbb357a4bedad235fe36b73fb1b5cdebb18c1e2\": rpc error: code = NotFound desc = could not find container \"edfb98ca40b9ca5d4d12e1054bbb357a4bedad235fe36b73fb1b5cdebb18c1e2\": container with ID starting with edfb98ca40b9ca5d4d12e1054bbb357a4bedad235fe36b73fb1b5cdebb18c1e2 not found: ID does not exist"
Jul 16 00:42:51.794252 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.794239    2563 scope.go:117] "RemoveContainer" containerID="73377ca2c9f5f98bdae8ac167ace8f28d9170705feff90bbff87dbf4058659ad"
Jul 16 00:42:51.794535 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.794500    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73377ca2c9f5f98bdae8ac167ace8f28d9170705feff90bbff87dbf4058659ad"} err="failed to get container status \"73377ca2c9f5f98bdae8ac167ace8f28d9170705feff90bbff87dbf4058659ad\": rpc error: code = NotFound desc = could not find container \"73377ca2c9f5f98bdae8ac167ace8f28d9170705feff90bbff87dbf4058659ad\": container with ID starting with 73377ca2c9f5f98bdae8ac167ace8f28d9170705feff90bbff87dbf4058659ad not found: ID does not exist"
Jul 16 00:42:51.794535 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.794525    2563 scope.go:117] "RemoveContainer" containerID="da5e0a6e27ccb609f6341ca1d7ae16b2c6b3054dc2a24bfc2145fe9ede975837"
Jul 16 00:42:51.795010 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.794917    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da5e0a6e27ccb609f6341ca1d7ae16b2c6b3054dc2a24bfc2145fe9ede975837"} err="failed to get container status \"da5e0a6e27ccb609f6341ca1d7ae16b2c6b3054dc2a24bfc2145fe9ede975837\": rpc error: code = NotFound desc = could not find container \"da5e0a6e27ccb609f6341ca1d7ae16b2c6b3054dc2a24bfc2145fe9ede975837\": container with ID starting with da5e0a6e27ccb609f6341ca1d7ae16b2c6b3054dc2a24bfc2145fe9ede975837 not found: ID does not exist"
Jul 16 00:42:51.795010 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.794941    2563 scope.go:117] "RemoveContainer" containerID="3b3d32f5e37e91811bbc20f3c1b425fa39f05f30266673ef09e240b586a20fd7"
Jul 16 00:42:51.795271 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.795244    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b3d32f5e37e91811bbc20f3c1b425fa39f05f30266673ef09e240b586a20fd7"} err="failed to get container status \"3b3d32f5e37e91811bbc20f3c1b425fa39f05f30266673ef09e240b586a20fd7\": rpc error: code = NotFound desc = could not find container \"3b3d32f5e37e91811bbc20f3c1b425fa39f05f30266673ef09e240b586a20fd7\": container with ID starting with 3b3d32f5e37e91811bbc20f3c1b425fa39f05f30266673ef09e240b586a20fd7 not found: ID does not exist"
Jul 16 00:42:51.795342 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.795273    2563 scope.go:117] "RemoveContainer" containerID="96c91fdd8e52ee998ebf81468edc2bb168994118fa5fa82ffe1361c2440d3d1e"
Jul 16 00:42:51.795625 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.795599    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96c91fdd8e52ee998ebf81468edc2bb168994118fa5fa82ffe1361c2440d3d1e"} err="failed to get container status \"96c91fdd8e52ee998ebf81468edc2bb168994118fa5fa82ffe1361c2440d3d1e\": rpc error: code = NotFound desc = could not find container \"96c91fdd8e52ee998ebf81468edc2bb168994118fa5fa82ffe1361c2440d3d1e\": container with ID starting with 96c91fdd8e52ee998ebf81468edc2bb168994118fa5fa82ffe1361c2440d3d1e not found: ID does not exist"
Jul 16 00:42:51.795732 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.795626    2563 scope.go:117] "RemoveContainer" containerID="602de56208d0126d303fe4020a5be610e107e35d39c452204f4aab86032c3710"
Jul 16 00:42:51.796383 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.796356    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"602de56208d0126d303fe4020a5be610e107e35d39c452204f4aab86032c3710"} err="failed to get container status \"602de56208d0126d303fe4020a5be610e107e35d39c452204f4aab86032c3710\": rpc error: code = NotFound desc = could not find container \"602de56208d0126d303fe4020a5be610e107e35d39c452204f4aab86032c3710\": container with ID starting with 602de56208d0126d303fe4020a5be610e107e35d39c452204f4aab86032c3710 not found: ID does not exist"
Jul 16 00:42:51.796469 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.796388    2563 scope.go:117] "RemoveContainer" containerID="233220bbd44752a0b45a1b01321a7b1d68e89cf5b75590206313f0afa56cc21f"
Jul 16 00:42:51.796469 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.796397    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 16 00:42:51.796635 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.796611    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"233220bbd44752a0b45a1b01321a7b1d68e89cf5b75590206313f0afa56cc21f"} err="failed to get container status \"233220bbd44752a0b45a1b01321a7b1d68e89cf5b75590206313f0afa56cc21f\": rpc error: code = NotFound desc = could not find container \"233220bbd44752a0b45a1b01321a7b1d68e89cf5b75590206313f0afa56cc21f\": container with ID starting with 233220bbd44752a0b45a1b01321a7b1d68e89cf5b75590206313f0afa56cc21f not found: ID does not exist"
Jul 16 00:42:51.796679 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.796637    2563 scope.go:117] "RemoveContainer" containerID="edfb98ca40b9ca5d4d12e1054bbb357a4bedad235fe36b73fb1b5cdebb18c1e2"
Jul 16 00:42:51.796718 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.796684    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d52ea21c-7d16-4007-a680-b67f3cfbcdc6" containerName="config-reloader"
Jul 16 00:42:51.796718 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.796696    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d52ea21c-7d16-4007-a680-b67f3cfbcdc6" containerName="config-reloader"
Jul 16 00:42:51.796718 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.796709    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d52ea21c-7d16-4007-a680-b67f3cfbcdc6" containerName="thanos-sidecar"
Jul 16 00:42:51.796718 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.796714    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d52ea21c-7d16-4007-a680-b67f3cfbcdc6" containerName="thanos-sidecar"
Jul 16 00:42:51.796884 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.796723    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d52ea21c-7d16-4007-a680-b67f3cfbcdc6" containerName="init-config-reloader"
Jul 16 00:42:51.796884 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.796728    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d52ea21c-7d16-4007-a680-b67f3cfbcdc6" containerName="init-config-reloader"
Jul 16 00:42:51.796884 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.796741    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d52ea21c-7d16-4007-a680-b67f3cfbcdc6" containerName="kube-rbac-proxy-thanos"
Jul 16 00:42:51.796884 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.796749    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d52ea21c-7d16-4007-a680-b67f3cfbcdc6" containerName="kube-rbac-proxy-thanos"
Jul 16 00:42:51.796884 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.796757    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d52ea21c-7d16-4007-a680-b67f3cfbcdc6" containerName="prometheus"
Jul 16 00:42:51.796884 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.796766    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d52ea21c-7d16-4007-a680-b67f3cfbcdc6" containerName="prometheus"
Jul 16 00:42:51.796884 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.796777    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d52ea21c-7d16-4007-a680-b67f3cfbcdc6" containerName="kube-rbac-proxy-web"
Jul 16 00:42:51.796884 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.796785    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d52ea21c-7d16-4007-a680-b67f3cfbcdc6" containerName="kube-rbac-proxy-web"
Jul 16 00:42:51.796884 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.796794    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d52ea21c-7d16-4007-a680-b67f3cfbcdc6" containerName="kube-rbac-proxy"
Jul 16 00:42:51.796884 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.796799    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d52ea21c-7d16-4007-a680-b67f3cfbcdc6" containerName="kube-rbac-proxy"
Jul 16 00:42:51.796884 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.796834    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edfb98ca40b9ca5d4d12e1054bbb357a4bedad235fe36b73fb1b5cdebb18c1e2"} err="failed to get container status \"edfb98ca40b9ca5d4d12e1054bbb357a4bedad235fe36b73fb1b5cdebb18c1e2\": rpc error: code = NotFound desc = could not find container \"edfb98ca40b9ca5d4d12e1054bbb357a4bedad235fe36b73fb1b5cdebb18c1e2\": container with ID starting with edfb98ca40b9ca5d4d12e1054bbb357a4bedad235fe36b73fb1b5cdebb18c1e2 not found: ID does not exist"
Jul 16 00:42:51.796884 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.796851    2563 scope.go:117] "RemoveContainer" containerID="73377ca2c9f5f98bdae8ac167ace8f28d9170705feff90bbff87dbf4058659ad"
Jul 16 00:42:51.796884 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.796844    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d52ea21c-7d16-4007-a680-b67f3cfbcdc6" containerName="kube-rbac-proxy"
Jul 16 00:42:51.797378 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.796896    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d52ea21c-7d16-4007-a680-b67f3cfbcdc6" containerName="thanos-sidecar"
Jul 16 00:42:51.797378 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.796902    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d52ea21c-7d16-4007-a680-b67f3cfbcdc6" containerName="prometheus"
Jul 16 00:42:51.797378 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.796909    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d52ea21c-7d16-4007-a680-b67f3cfbcdc6" containerName="kube-rbac-proxy-thanos"
Jul 16 00:42:51.797378 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.796914    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d52ea21c-7d16-4007-a680-b67f3cfbcdc6" containerName="kube-rbac-proxy-web"
Jul 16 00:42:51.797378 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.796921    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d52ea21c-7d16-4007-a680-b67f3cfbcdc6" containerName="config-reloader"
Jul 16 00:42:51.797378 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.797064    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73377ca2c9f5f98bdae8ac167ace8f28d9170705feff90bbff87dbf4058659ad"} err="failed to get container status \"73377ca2c9f5f98bdae8ac167ace8f28d9170705feff90bbff87dbf4058659ad\": rpc error: code = NotFound desc = could not find container \"73377ca2c9f5f98bdae8ac167ace8f28d9170705feff90bbff87dbf4058659ad\": container with ID starting with 73377ca2c9f5f98bdae8ac167ace8f28d9170705feff90bbff87dbf4058659ad not found: ID does not exist"
Jul 16 00:42:51.797378 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.797078    2563 scope.go:117] "RemoveContainer" containerID="da5e0a6e27ccb609f6341ca1d7ae16b2c6b3054dc2a24bfc2145fe9ede975837"
Jul 16 00:42:51.797378 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.797273    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da5e0a6e27ccb609f6341ca1d7ae16b2c6b3054dc2a24bfc2145fe9ede975837"} err="failed to get container status \"da5e0a6e27ccb609f6341ca1d7ae16b2c6b3054dc2a24bfc2145fe9ede975837\": rpc error: code = NotFound desc = could not find container \"da5e0a6e27ccb609f6341ca1d7ae16b2c6b3054dc2a24bfc2145fe9ede975837\": container with ID starting with da5e0a6e27ccb609f6341ca1d7ae16b2c6b3054dc2a24bfc2145fe9ede975837 not found: ID does not exist"
Jul 16 00:42:51.797378 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.797297    2563 scope.go:117] "RemoveContainer" containerID="3b3d32f5e37e91811bbc20f3c1b425fa39f05f30266673ef09e240b586a20fd7"
Jul 16 00:42:51.797764 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.797526    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b3d32f5e37e91811bbc20f3c1b425fa39f05f30266673ef09e240b586a20fd7"} err="failed to get container status \"3b3d32f5e37e91811bbc20f3c1b425fa39f05f30266673ef09e240b586a20fd7\": rpc error: code = NotFound desc = could not find container \"3b3d32f5e37e91811bbc20f3c1b425fa39f05f30266673ef09e240b586a20fd7\": container with ID starting with 3b3d32f5e37e91811bbc20f3c1b425fa39f05f30266673ef09e240b586a20fd7 not found: ID does not exist"
Jul 16 00:42:51.797764 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.797541    2563 scope.go:117] "RemoveContainer" containerID="96c91fdd8e52ee998ebf81468edc2bb168994118fa5fa82ffe1361c2440d3d1e"
Jul 16 00:42:51.797830 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.797805    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96c91fdd8e52ee998ebf81468edc2bb168994118fa5fa82ffe1361c2440d3d1e"} err="failed to get container status \"96c91fdd8e52ee998ebf81468edc2bb168994118fa5fa82ffe1361c2440d3d1e\": rpc error: code = NotFound desc = could not find container \"96c91fdd8e52ee998ebf81468edc2bb168994118fa5fa82ffe1361c2440d3d1e\": container with ID starting with 96c91fdd8e52ee998ebf81468edc2bb168994118fa5fa82ffe1361c2440d3d1e not found: ID does not exist"
Jul 16 00:42:51.797830 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.797821    2563 scope.go:117] "RemoveContainer" containerID="602de56208d0126d303fe4020a5be610e107e35d39c452204f4aab86032c3710"
Jul 16 00:42:51.798040 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.798023    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"602de56208d0126d303fe4020a5be610e107e35d39c452204f4aab86032c3710"} err="failed to get container status \"602de56208d0126d303fe4020a5be610e107e35d39c452204f4aab86032c3710\": rpc error: code = NotFound desc = could not find container \"602de56208d0126d303fe4020a5be610e107e35d39c452204f4aab86032c3710\": container with ID starting with 602de56208d0126d303fe4020a5be610e107e35d39c452204f4aab86032c3710 not found: ID does not exist"
Jul 16 00:42:51.802201 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.802186    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 00:42:51.804267 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.804240    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 16 00:42:51.804344 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.804283    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 16 00:42:51.804954 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.804938    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 16 00:42:51.805073 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.804966    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-x4z5j\""
Jul 16 00:42:51.805073 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.805014    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 16 00:42:51.805196 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.805096    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 16 00:42:51.805256 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.805200    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 16 00:42:51.805256 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.805205    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 16 00:42:51.805533 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.805515    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 16 00:42:51.805627 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.805555    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-3oml8al8okbbk\""
Jul 16 00:42:51.805762 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.805738    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 16 00:42:51.805849 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.805737    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 16 00:42:51.805849 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.805812    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 16 00:42:51.806619 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.806601    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 16 00:42:51.808767 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.808748    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 16 00:42:51.810845 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.810828    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 16 00:42:51.816433 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.816413    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 16 00:42:51.869621 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.869594    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ltxf\" (UniqueName: \"kubernetes.io/projected/c888296a-9f5e-47a1-ad37-985053642fc6-kube-api-access-9ltxf\") pod \"prometheus-k8s-0\" (UID: \"c888296a-9f5e-47a1-ad37-985053642fc6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 00:42:51.869759 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.869634    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c888296a-9f5e-47a1-ad37-985053642fc6-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c888296a-9f5e-47a1-ad37-985053642fc6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 00:42:51.869759 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.869657    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c888296a-9f5e-47a1-ad37-985053642fc6-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c888296a-9f5e-47a1-ad37-985053642fc6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 00:42:51.869833 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.869753    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c888296a-9f5e-47a1-ad37-985053642fc6-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c888296a-9f5e-47a1-ad37-985053642fc6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 00:42:51.869833 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.869787    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c888296a-9f5e-47a1-ad37-985053642fc6-config-out\") pod \"prometheus-k8s-0\" (UID: \"c888296a-9f5e-47a1-ad37-985053642fc6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 00:42:51.869833 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.869806    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c888296a-9f5e-47a1-ad37-985053642fc6-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c888296a-9f5e-47a1-ad37-985053642fc6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 00:42:51.869833 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.869826    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c888296a-9f5e-47a1-ad37-985053642fc6-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c888296a-9f5e-47a1-ad37-985053642fc6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 00:42:51.870052 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.869876    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c888296a-9f5e-47a1-ad37-985053642fc6-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c888296a-9f5e-47a1-ad37-985053642fc6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 00:42:51.870052 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.869903    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c888296a-9f5e-47a1-ad37-985053642fc6-web-config\") pod \"prometheus-k8s-0\" (UID: \"c888296a-9f5e-47a1-ad37-985053642fc6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 00:42:51.870052 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.869940    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c888296a-9f5e-47a1-ad37-985053642fc6-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c888296a-9f5e-47a1-ad37-985053642fc6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 00:42:51.870052 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.869990    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c888296a-9f5e-47a1-ad37-985053642fc6-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c888296a-9f5e-47a1-ad37-985053642fc6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 00:42:51.870052 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.870017    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c888296a-9f5e-47a1-ad37-985053642fc6-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c888296a-9f5e-47a1-ad37-985053642fc6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 00:42:51.870052 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.870042    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c888296a-9f5e-47a1-ad37-985053642fc6-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c888296a-9f5e-47a1-ad37-985053642fc6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 00:42:51.870336 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.870068    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c888296a-9f5e-47a1-ad37-985053642fc6-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c888296a-9f5e-47a1-ad37-985053642fc6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 00:42:51.870336 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.870094    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c888296a-9f5e-47a1-ad37-985053642fc6-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c888296a-9f5e-47a1-ad37-985053642fc6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 00:42:51.870336 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.870131    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c888296a-9f5e-47a1-ad37-985053642fc6-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c888296a-9f5e-47a1-ad37-985053642fc6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 00:42:51.870336 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.870172    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/c888296a-9f5e-47a1-ad37-985053642fc6-config\") pod \"prometheus-k8s-0\" (UID: \"c888296a-9f5e-47a1-ad37-985053642fc6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 00:42:51.870336 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.870201    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c888296a-9f5e-47a1-ad37-985053642fc6-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c888296a-9f5e-47a1-ad37-985053642fc6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 00:42:51.870336 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.870236    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c888296a-9f5e-47a1-ad37-985053642fc6-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c888296a-9f5e-47a1-ad37-985053642fc6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 00:42:51.870336 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.870277    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c888296a-9f5e-47a1-ad37-985053642fc6-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c888296a-9f5e-47a1-ad37-985053642fc6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 00:42:51.970645 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.970603    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c888296a-9f5e-47a1-ad37-985053642fc6-config-out\") pod \"prometheus-k8s-0\" (UID: \"c888296a-9f5e-47a1-ad37-985053642fc6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 00:42:51.970645 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.970650    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c888296a-9f5e-47a1-ad37-985053642fc6-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c888296a-9f5e-47a1-ad37-985053642fc6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 00:42:51.970837 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.970673    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c888296a-9f5e-47a1-ad37-985053642fc6-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c888296a-9f5e-47a1-ad37-985053642fc6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 00:42:51.970837 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.970722    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c888296a-9f5e-47a1-ad37-985053642fc6-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c888296a-9f5e-47a1-ad37-985053642fc6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 00:42:51.970837 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.970754    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c888296a-9f5e-47a1-ad37-985053642fc6-web-config\") pod \"prometheus-k8s-0\" (UID: \"c888296a-9f5e-47a1-ad37-985053642fc6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 00:42:51.970837 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.970790    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c888296a-9f5e-47a1-ad37-985053642fc6-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c888296a-9f5e-47a1-ad37-985053642fc6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 00:42:51.970837 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.970827    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c888296a-9f5e-47a1-ad37-985053642fc6-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c888296a-9f5e-47a1-ad37-985053642fc6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 00:42:51.971067 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.970852    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c888296a-9f5e-47a1-ad37-985053642fc6-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c888296a-9f5e-47a1-ad37-985053642fc6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 00:42:51.971067 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.970896    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c888296a-9f5e-47a1-ad37-985053642fc6-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c888296a-9f5e-47a1-ad37-985053642fc6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 00:42:51.971067 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.970920    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c888296a-9f5e-47a1-ad37-985053642fc6-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c888296a-9f5e-47a1-ad37-985053642fc6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 00:42:51.971067 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.970935    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c888296a-9f5e-47a1-ad37-985053642fc6-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c888296a-9f5e-47a1-ad37-985053642fc6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 00:42:51.971067 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.970952    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c888296a-9f5e-47a1-ad37-985053642fc6-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c888296a-9f5e-47a1-ad37-985053642fc6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 00:42:51.971067 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.970970    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/c888296a-9f5e-47a1-ad37-985053642fc6-config\") pod \"prometheus-k8s-0\" (UID: \"c888296a-9f5e-47a1-ad37-985053642fc6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 00:42:51.971067 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.971000    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c888296a-9f5e-47a1-ad37-985053642fc6-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c888296a-9f5e-47a1-ad37-985053642fc6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 00:42:51.971067 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.971025    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c888296a-9f5e-47a1-ad37-985053642fc6-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c888296a-9f5e-47a1-ad37-985053642fc6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 00:42:51.971067 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.971055    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c888296a-9f5e-47a1-ad37-985053642fc6-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c888296a-9f5e-47a1-ad37-985053642fc6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 00:42:51.971369 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.971079    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ltxf\" (UniqueName: \"kubernetes.io/projected/c888296a-9f5e-47a1-ad37-985053642fc6-kube-api-access-9ltxf\") pod \"prometheus-k8s-0\" (UID: \"c888296a-9f5e-47a1-ad37-985053642fc6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 00:42:51.971369 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.971108    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c888296a-9f5e-47a1-ad37-985053642fc6-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c888296a-9f5e-47a1-ad37-985053642fc6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 00:42:51.971369 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.971122    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c888296a-9f5e-47a1-ad37-985053642fc6-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c888296a-9f5e-47a1-ad37-985053642fc6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 00:42:51.971369 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.971159    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c888296a-9f5e-47a1-ad37-985053642fc6-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c888296a-9f5e-47a1-ad37-985053642fc6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 00:42:51.972205 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.972176    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c888296a-9f5e-47a1-ad37-985053642fc6-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c888296a-9f5e-47a1-ad37-985053642fc6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 00:42:51.973248 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.972923    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c888296a-9f5e-47a1-ad37-985053642fc6-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c888296a-9f5e-47a1-ad37-985053642fc6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 00:42:51.974087 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.973472    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c888296a-9f5e-47a1-ad37-985053642fc6-config-out\") pod \"prometheus-k8s-0\" (UID: \"c888296a-9f5e-47a1-ad37-985053642fc6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 00:42:51.974087 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.973744    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c888296a-9f5e-47a1-ad37-985053642fc6-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c888296a-9f5e-47a1-ad37-985053642fc6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 00:42:51.974087 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.973826    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c888296a-9f5e-47a1-ad37-985053642fc6-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c888296a-9f5e-47a1-ad37-985053642fc6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 00:42:51.974087 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.973939    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c888296a-9f5e-47a1-ad37-985053642fc6-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c888296a-9f5e-47a1-ad37-985053642fc6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 00:42:51.974087 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.974040    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c888296a-9f5e-47a1-ad37-985053642fc6-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c888296a-9f5e-47a1-ad37-985053642fc6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 00:42:51.975092 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.974317    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c888296a-9f5e-47a1-ad37-985053642fc6-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c888296a-9f5e-47a1-ad37-985053642fc6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 00:42:51.975092 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.974402    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c888296a-9f5e-47a1-ad37-985053642fc6-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c888296a-9f5e-47a1-ad37-985053642fc6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 00:42:51.975092 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.974462    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c888296a-9f5e-47a1-ad37-985053642fc6-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c888296a-9f5e-47a1-ad37-985053642fc6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 00:42:51.975092 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.974598    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c888296a-9f5e-47a1-ad37-985053642fc6-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c888296a-9f5e-47a1-ad37-985053642fc6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 00:42:51.975092 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.974679    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c888296a-9f5e-47a1-ad37-985053642fc6-web-config\") pod \"prometheus-k8s-0\" (UID: \"c888296a-9f5e-47a1-ad37-985053642fc6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 00:42:51.975092 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.974830    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c888296a-9f5e-47a1-ad37-985053642fc6-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c888296a-9f5e-47a1-ad37-985053642fc6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 00:42:51.975842 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.975817    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c888296a-9f5e-47a1-ad37-985053642fc6-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c888296a-9f5e-47a1-ad37-985053642fc6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 00:42:51.976178 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.976157    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c888296a-9f5e-47a1-ad37-985053642fc6-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c888296a-9f5e-47a1-ad37-985053642fc6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 00:42:51.977003 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.976958    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c888296a-9f5e-47a1-ad37-985053642fc6-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c888296a-9f5e-47a1-ad37-985053642fc6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 00:42:51.977062 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.977010    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/c888296a-9f5e-47a1-ad37-985053642fc6-config\") pod \"prometheus-k8s-0\" (UID: \"c888296a-9f5e-47a1-ad37-985053642fc6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 00:42:51.977062 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.977039    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c888296a-9f5e-47a1-ad37-985053642fc6-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c888296a-9f5e-47a1-ad37-985053642fc6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 00:42:51.977133 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.977113    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c888296a-9f5e-47a1-ad37-985053642fc6-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c888296a-9f5e-47a1-ad37-985053642fc6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 00:42:51.983103 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:51.983081    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ltxf\" (UniqueName: \"kubernetes.io/projected/c888296a-9f5e-47a1-ad37-985053642fc6-kube-api-access-9ltxf\") pod \"prometheus-k8s-0\" (UID: \"c888296a-9f5e-47a1-ad37-985053642fc6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 00:42:52.005591 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:52.005561    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d52ea21c-7d16-4007-a680-b67f3cfbcdc6" path="/var/lib/kubelet/pods/d52ea21c-7d16-4007-a680-b67f3cfbcdc6/volumes"
Jul 16 00:42:52.112667 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:52.112635    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 00:42:52.247144 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:52.247117    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 16 00:42:52.249429 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:42:52.249402    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc888296a_9f5e_47a1_ad37_985053642fc6.slice/crio-948055bcae4037deb80441dfd785aa085a1b3fe1037ee174fc2db9882d541fdf WatchSource:0}: Error finding container 948055bcae4037deb80441dfd785aa085a1b3fe1037ee174fc2db9882d541fdf: Status 404 returned error can't find the container with id 948055bcae4037deb80441dfd785aa085a1b3fe1037ee174fc2db9882d541fdf
Jul 16 00:42:52.732774 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:52.732746    2563 generic.go:356] "Generic (PLEG): container finished" podID="c888296a-9f5e-47a1-ad37-985053642fc6" containerID="80e5dd506e2738ed6def7a7b25884c5c29c49d64d9b10f8e20d17b7a64bcf98b" exitCode=0
Jul 16 00:42:52.732963 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:52.732804    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c888296a-9f5e-47a1-ad37-985053642fc6","Type":"ContainerDied","Data":"80e5dd506e2738ed6def7a7b25884c5c29c49d64d9b10f8e20d17b7a64bcf98b"}
Jul 16 00:42:52.732963 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:52.732880    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c888296a-9f5e-47a1-ad37-985053642fc6","Type":"ContainerStarted","Data":"948055bcae4037deb80441dfd785aa085a1b3fe1037ee174fc2db9882d541fdf"}
Jul 16 00:42:53.740513 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:53.740478    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c888296a-9f5e-47a1-ad37-985053642fc6","Type":"ContainerStarted","Data":"c6db94dc45ae6efbf0a5009fb6080596b1c9725ce53e2da23ce95ed6a83afee9"}
Jul 16 00:42:53.740990 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:53.740584    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c888296a-9f5e-47a1-ad37-985053642fc6","Type":"ContainerStarted","Data":"b72f5d337739c5a7ae49432b1752f53f0b7e9b2a821853c19b4fd70b43e2fe12"}
Jul 16 00:42:53.740990 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:53.740636    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c888296a-9f5e-47a1-ad37-985053642fc6","Type":"ContainerStarted","Data":"f85c4dc32a09799fa65181822c9233857ba28baf3c4cee35db25db11b3bdab35"}
Jul 16 00:42:53.740990 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:53.740681    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c888296a-9f5e-47a1-ad37-985053642fc6","Type":"ContainerStarted","Data":"e205613c4a8ac25d76cf7b003ef5d64f2b6034170f6a462f3e0850ad89643592"}
Jul 16 00:42:53.740990 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:53.740721    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c888296a-9f5e-47a1-ad37-985053642fc6","Type":"ContainerStarted","Data":"5b577a87a4fcb6c2bca1407bf77f10cbaec929891f7687b03686785f3f673b73"}
Jul 16 00:42:53.740990 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:53.740778    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c888296a-9f5e-47a1-ad37-985053642fc6","Type":"ContainerStarted","Data":"3ccbe8eb21d8cd6201befc0972bcad59f9beb8b7f6c4bae426c72e866c837efe"}
Jul 16 00:42:53.771581 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:53.771517    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.749447071 podStartE2EDuration="2.771501474s" totalImagesPullingTime="0s" totalInitContainerRuntime="22.054403ms" isStatefulPod=false podCreationTimestamp="2026-07-16 00:42:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 00:42:53.77034865 +0000 UTC m=+146.205610388" watchObservedRunningTime="2026-07-16 00:42:53.771501474 +0000 UTC m=+146.206763204"
Jul 16 00:42:57.113053 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:42:57.113008    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 00:43:52.113819 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:43:52.113781    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 00:43:52.129034 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:43:52.129009    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 00:43:52.972808 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:43:52.972783    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 00:45:27.858539 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:45:27.858494    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bn5hr_e990a21d-614b-4d1d-be54-0df6e44487ed/ovn-acl-logging/0.log"
Jul 16 00:45:27.860633 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:45:27.860612    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bn5hr_e990a21d-614b-4d1d-be54-0df6e44487ed/ovn-acl-logging/0.log"
Jul 16 00:45:27.862333 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:45:27.862315    2563 kubelet.go:1648] "Image garbage collection succeeded"
Jul 16 00:45:46.712180 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:45:46.712143    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-q58xz"]
Jul 16 00:45:46.715513 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:45:46.715497    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-q58xz"
Jul 16 00:45:46.717301 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:45:46.717280    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 16 00:45:46.717800 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:45:46.717786    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 16 00:45:46.717873 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:45:46.717819    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-2drlg\""
Jul 16 00:45:46.723502 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:45:46.723471    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-q58xz"]
Jul 16 00:45:46.752275 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:45:46.752241    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mfhr\" (UniqueName: \"kubernetes.io/projected/8b4ddb9a-aabe-424a-a646-73c81269deb7-kube-api-access-6mfhr\") pod \"openshift-lws-operator-fd8ccff4c-q58xz\" (UID: \"8b4ddb9a-aabe-424a-a646-73c81269deb7\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-q58xz"
Jul 16 00:45:46.752275 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:45:46.752274    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8b4ddb9a-aabe-424a-a646-73c81269deb7-tmp\") pod \"openshift-lws-operator-fd8ccff4c-q58xz\" (UID: \"8b4ddb9a-aabe-424a-a646-73c81269deb7\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-q58xz"
Jul 16 00:45:46.852956 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:45:46.852918    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mfhr\" (UniqueName: \"kubernetes.io/projected/8b4ddb9a-aabe-424a-a646-73c81269deb7-kube-api-access-6mfhr\") pod \"openshift-lws-operator-fd8ccff4c-q58xz\" (UID: \"8b4ddb9a-aabe-424a-a646-73c81269deb7\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-q58xz"
Jul 16 00:45:46.852956 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:45:46.852957    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8b4ddb9a-aabe-424a-a646-73c81269deb7-tmp\") pod \"openshift-lws-operator-fd8ccff4c-q58xz\" (UID: \"8b4ddb9a-aabe-424a-a646-73c81269deb7\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-q58xz"
Jul 16 00:45:46.853295 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:45:46.853280    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8b4ddb9a-aabe-424a-a646-73c81269deb7-tmp\") pod \"openshift-lws-operator-fd8ccff4c-q58xz\" (UID: \"8b4ddb9a-aabe-424a-a646-73c81269deb7\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-q58xz"
Jul 16 00:45:46.862301 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:45:46.862277    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mfhr\" (UniqueName: \"kubernetes.io/projected/8b4ddb9a-aabe-424a-a646-73c81269deb7-kube-api-access-6mfhr\") pod \"openshift-lws-operator-fd8ccff4c-q58xz\" (UID: \"8b4ddb9a-aabe-424a-a646-73c81269deb7\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-q58xz"
Jul 16 00:45:47.037434 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:45:47.037352    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-q58xz"
Jul 16 00:45:47.166619 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:45:47.166591    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-q58xz"]
Jul 16 00:45:47.169046 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:45:47.169017    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8b4ddb9a_aabe_424a_a646_73c81269deb7.slice/crio-ce743e32ffe427056e94cbae720259cd05c8d1d4b1b1434b633d5db92095f396 WatchSource:0}: Error finding container ce743e32ffe427056e94cbae720259cd05c8d1d4b1b1434b633d5db92095f396: Status 404 returned error can't find the container with id ce743e32ffe427056e94cbae720259cd05c8d1d4b1b1434b633d5db92095f396
Jul 16 00:45:47.170673 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:45:47.170652    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 16 00:45:47.361144 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:45:47.361100    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-q58xz" event={"ID":"8b4ddb9a-aabe-424a-a646-73c81269deb7","Type":"ContainerStarted","Data":"ce743e32ffe427056e94cbae720259cd05c8d1d4b1b1434b633d5db92095f396"}
Jul 16 00:45:50.374650 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:45:50.374616    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-q58xz" event={"ID":"8b4ddb9a-aabe-424a-a646-73c81269deb7","Type":"ContainerStarted","Data":"5990cd012182ab537b16390479521421d4b1bbabfa8b0c561eceb374e3ac996a"}
Jul 16 00:46:01.354994 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:46:01.354929    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-q58xz" podStartSLOduration=12.590723009 podStartE2EDuration="15.354905953s" totalImagesPullingTime="2.764182944s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 00:45:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 00:45:50.389434806 +0000 UTC m=+322.824696547" watchObservedRunningTime="2026-07-16 00:46:01.354905953 +0000 UTC m=+333.790167691"
Jul 16 00:46:01.355452 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:46:01.355061    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-c857fdb5b-5bxrx"]
Jul 16 00:46:01.358475 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:46:01.358455    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-c857fdb5b-5bxrx"
Jul 16 00:46:01.362039 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:46:01.362012    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-64qvf\""
Jul 16 00:46:01.362169 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:46:01.362066    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 16 00:46:01.362169 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:46:01.362015    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 16 00:46:01.362270 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:46:01.362143    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 16 00:46:01.377222 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:46:01.377193    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-c857fdb5b-5bxrx"]
Jul 16 00:46:01.479373 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:46:01.479322    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/70e026c5-9607-43bd-b482-8555fb4f6c41-cert\") pod \"lws-controller-manager-c857fdb5b-5bxrx\" (UID: \"70e026c5-9607-43bd-b482-8555fb4f6c41\") " pod="openshift-lws-operator/lws-controller-manager-c857fdb5b-5bxrx"
Jul 16 00:46:01.479553 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:46:01.479406    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24l2x\" (UniqueName: \"kubernetes.io/projected/70e026c5-9607-43bd-b482-8555fb4f6c41-kube-api-access-24l2x\") pod \"lws-controller-manager-c857fdb5b-5bxrx\" (UID: \"70e026c5-9607-43bd-b482-8555fb4f6c41\") " pod="openshift-lws-operator/lws-controller-manager-c857fdb5b-5bxrx"
Jul 16 00:46:01.479553 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:46:01.479431    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/70e026c5-9607-43bd-b482-8555fb4f6c41-manager-config\") pod \"lws-controller-manager-c857fdb5b-5bxrx\" (UID: \"70e026c5-9607-43bd-b482-8555fb4f6c41\") " pod="openshift-lws-operator/lws-controller-manager-c857fdb5b-5bxrx"
Jul 16 00:46:01.479553 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:46:01.479453    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/70e026c5-9607-43bd-b482-8555fb4f6c41-metrics-cert\") pod \"lws-controller-manager-c857fdb5b-5bxrx\" (UID: \"70e026c5-9607-43bd-b482-8555fb4f6c41\") " pod="openshift-lws-operator/lws-controller-manager-c857fdb5b-5bxrx"
Jul 16 00:46:01.580629 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:46:01.580601    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/70e026c5-9607-43bd-b482-8555fb4f6c41-metrics-cert\") pod \"lws-controller-manager-c857fdb5b-5bxrx\" (UID: \"70e026c5-9607-43bd-b482-8555fb4f6c41\") " pod="openshift-lws-operator/lws-controller-manager-c857fdb5b-5bxrx"
Jul 16 00:46:01.580743 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:46:01.580676    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/70e026c5-9607-43bd-b482-8555fb4f6c41-cert\") pod \"lws-controller-manager-c857fdb5b-5bxrx\" (UID: \"70e026c5-9607-43bd-b482-8555fb4f6c41\") " pod="openshift-lws-operator/lws-controller-manager-c857fdb5b-5bxrx"
Jul 16 00:46:01.580743 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:46:01.580715    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24l2x\" (UniqueName: \"kubernetes.io/projected/70e026c5-9607-43bd-b482-8555fb4f6c41-kube-api-access-24l2x\") pod \"lws-controller-manager-c857fdb5b-5bxrx\" (UID: \"70e026c5-9607-43bd-b482-8555fb4f6c41\") " pod="openshift-lws-operator/lws-controller-manager-c857fdb5b-5bxrx"
Jul 16 00:46:01.580837 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:46:01.580744    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/70e026c5-9607-43bd-b482-8555fb4f6c41-manager-config\") pod \"lws-controller-manager-c857fdb5b-5bxrx\" (UID: \"70e026c5-9607-43bd-b482-8555fb4f6c41\") " pod="openshift-lws-operator/lws-controller-manager-c857fdb5b-5bxrx"
Jul 16 00:46:01.581430 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:46:01.581402    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/70e026c5-9607-43bd-b482-8555fb4f6c41-manager-config\") pod \"lws-controller-manager-c857fdb5b-5bxrx\" (UID: \"70e026c5-9607-43bd-b482-8555fb4f6c41\") " pod="openshift-lws-operator/lws-controller-manager-c857fdb5b-5bxrx"
Jul 16 00:46:01.583146 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:46:01.583129    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/70e026c5-9607-43bd-b482-8555fb4f6c41-metrics-cert\") pod \"lws-controller-manager-c857fdb5b-5bxrx\" (UID: \"70e026c5-9607-43bd-b482-8555fb4f6c41\") " pod="openshift-lws-operator/lws-controller-manager-c857fdb5b-5bxrx"
Jul 16 00:46:01.583146 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:46:01.583138    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/70e026c5-9607-43bd-b482-8555fb4f6c41-cert\") pod \"lws-controller-manager-c857fdb5b-5bxrx\" (UID: \"70e026c5-9607-43bd-b482-8555fb4f6c41\") " pod="openshift-lws-operator/lws-controller-manager-c857fdb5b-5bxrx"
Jul 16 00:46:01.588485 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:46:01.588456    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-24l2x\" (UniqueName: \"kubernetes.io/projected/70e026c5-9607-43bd-b482-8555fb4f6c41-kube-api-access-24l2x\") pod \"lws-controller-manager-c857fdb5b-5bxrx\" (UID: \"70e026c5-9607-43bd-b482-8555fb4f6c41\") " pod="openshift-lws-operator/lws-controller-manager-c857fdb5b-5bxrx"
Jul 16 00:46:01.667457 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:46:01.667369    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-c857fdb5b-5bxrx"
Jul 16 00:46:01.824818 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:46:01.824793    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-c857fdb5b-5bxrx"]
Jul 16 00:46:01.826805 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:46:01.826775    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70e026c5_9607_43bd_b482_8555fb4f6c41.slice/crio-31a112a37b30fcfa1626ce6f2ed46883b70b8ff1cc892069649b3364e89c70b5 WatchSource:0}: Error finding container 31a112a37b30fcfa1626ce6f2ed46883b70b8ff1cc892069649b3364e89c70b5: Status 404 returned error can't find the container with id 31a112a37b30fcfa1626ce6f2ed46883b70b8ff1cc892069649b3364e89c70b5
Jul 16 00:46:02.423080 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:46:02.423029    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-c857fdb5b-5bxrx" event={"ID":"70e026c5-9607-43bd-b482-8555fb4f6c41","Type":"ContainerStarted","Data":"31a112a37b30fcfa1626ce6f2ed46883b70b8ff1cc892069649b3364e89c70b5"}
Jul 16 00:46:04.433839 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:46:04.433797    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-c857fdb5b-5bxrx" event={"ID":"70e026c5-9607-43bd-b482-8555fb4f6c41","Type":"ContainerStarted","Data":"e263f685bfc5c8c6710eaf8a36bfa6c709253a6e36e0cf83b036b39c1978ed3a"}
Jul 16 00:46:04.434299 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:46:04.433890    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-c857fdb5b-5bxrx"
Jul 16 00:46:04.453118 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:46:04.453015    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-c857fdb5b-5bxrx" podStartSLOduration=1.091766331 podStartE2EDuration="3.452994673s" totalImagesPullingTime="2.361228342s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 00:46:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 00:46:04.451351786 +0000 UTC m=+336.886613527" watchObservedRunningTime="2026-07-16 00:46:04.452994673 +0000 UTC m=+336.888256413"
Jul 16 00:46:09.590483 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:46:09.590447    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5cf8c55d45-8d6hh"]
Jul 16 00:46:09.597515 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:46:09.597482    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5cf8c55d45-8d6hh"
Jul 16 00:46:09.600108 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:46:09.600084    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 16 00:46:09.600331 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:46:09.600314    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 16 00:46:09.600417 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:46:09.600401    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 16 00:46:09.600491 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:46:09.600402    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 16 00:46:09.604106 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:46:09.603816    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-9kszx\""
Jul 16 00:46:09.616034 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:46:09.616007    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5cf8c55d45-8d6hh"]
Jul 16 00:46:09.750505 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:46:09.750464    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l69ts\" (UniqueName: \"kubernetes.io/projected/c9446ea3-dac9-4a1a-9ad3-1fda85d9f151-kube-api-access-l69ts\") pod \"opendatahub-operator-controller-manager-5cf8c55d45-8d6hh\" (UID: \"c9446ea3-dac9-4a1a-9ad3-1fda85d9f151\") " pod="opendatahub/opendatahub-operator-controller-manager-5cf8c55d45-8d6hh"
Jul 16 00:46:09.750505 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:46:09.750509    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c9446ea3-dac9-4a1a-9ad3-1fda85d9f151-manifests\") pod \"opendatahub-operator-controller-manager-5cf8c55d45-8d6hh\" (UID: \"c9446ea3-dac9-4a1a-9ad3-1fda85d9f151\") " pod="opendatahub/opendatahub-operator-controller-manager-5cf8c55d45-8d6hh"
Jul 16 00:46:09.750718 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:46:09.750554    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c9446ea3-dac9-4a1a-9ad3-1fda85d9f151-tmp\") pod \"opendatahub-operator-controller-manager-5cf8c55d45-8d6hh\" (UID: \"c9446ea3-dac9-4a1a-9ad3-1fda85d9f151\") " pod="opendatahub/opendatahub-operator-controller-manager-5cf8c55d45-8d6hh"
Jul 16 00:46:09.750718 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:46:09.750586    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c9446ea3-dac9-4a1a-9ad3-1fda85d9f151-webhook-cert\") pod \"opendatahub-operator-controller-manager-5cf8c55d45-8d6hh\" (UID: \"c9446ea3-dac9-4a1a-9ad3-1fda85d9f151\") " pod="opendatahub/opendatahub-operator-controller-manager-5cf8c55d45-8d6hh"
Jul 16 00:46:09.750718 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:46:09.750671    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c9446ea3-dac9-4a1a-9ad3-1fda85d9f151-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5cf8c55d45-8d6hh\" (UID: \"c9446ea3-dac9-4a1a-9ad3-1fda85d9f151\") " pod="opendatahub/opendatahub-operator-controller-manager-5cf8c55d45-8d6hh"
Jul 16 00:46:09.851254 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:46:09.851167    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c9446ea3-dac9-4a1a-9ad3-1fda85d9f151-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5cf8c55d45-8d6hh\" (UID: \"c9446ea3-dac9-4a1a-9ad3-1fda85d9f151\") " pod="opendatahub/opendatahub-operator-controller-manager-5cf8c55d45-8d6hh"
Jul 16 00:46:09.851254 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:46:09.851227    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l69ts\" (UniqueName: \"kubernetes.io/projected/c9446ea3-dac9-4a1a-9ad3-1fda85d9f151-kube-api-access-l69ts\") pod \"opendatahub-operator-controller-manager-5cf8c55d45-8d6hh\" (UID: \"c9446ea3-dac9-4a1a-9ad3-1fda85d9f151\") " pod="opendatahub/opendatahub-operator-controller-manager-5cf8c55d45-8d6hh"
Jul 16 00:46:09.851254 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:46:09.851253    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c9446ea3-dac9-4a1a-9ad3-1fda85d9f151-manifests\") pod \"opendatahub-operator-controller-manager-5cf8c55d45-8d6hh\" (UID: \"c9446ea3-dac9-4a1a-9ad3-1fda85d9f151\") " pod="opendatahub/opendatahub-operator-controller-manager-5cf8c55d45-8d6hh"
Jul 16 00:46:09.851492 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:46:09.851298    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c9446ea3-dac9-4a1a-9ad3-1fda85d9f151-tmp\") pod \"opendatahub-operator-controller-manager-5cf8c55d45-8d6hh\" (UID: \"c9446ea3-dac9-4a1a-9ad3-1fda85d9f151\") " pod="opendatahub/opendatahub-operator-controller-manager-5cf8c55d45-8d6hh"
Jul 16 00:46:09.851492 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:46:09.851323    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c9446ea3-dac9-4a1a-9ad3-1fda85d9f151-webhook-cert\") pod \"opendatahub-operator-controller-manager-5cf8c55d45-8d6hh\" (UID: \"c9446ea3-dac9-4a1a-9ad3-1fda85d9f151\") " pod="opendatahub/opendatahub-operator-controller-manager-5cf8c55d45-8d6hh"
Jul 16 00:46:09.851733 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:46:09.851703    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c9446ea3-dac9-4a1a-9ad3-1fda85d9f151-manifests\") pod \"opendatahub-operator-controller-manager-5cf8c55d45-8d6hh\" (UID: \"c9446ea3-dac9-4a1a-9ad3-1fda85d9f151\") " pod="opendatahub/opendatahub-operator-controller-manager-5cf8c55d45-8d6hh"
Jul 16 00:46:09.853686 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:46:09.853663    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c9446ea3-dac9-4a1a-9ad3-1fda85d9f151-tmp\") pod \"opendatahub-operator-controller-manager-5cf8c55d45-8d6hh\" (UID: \"c9446ea3-dac9-4a1a-9ad3-1fda85d9f151\") " pod="opendatahub/opendatahub-operator-controller-manager-5cf8c55d45-8d6hh"
Jul 16 00:46:09.853772 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:46:09.853751    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c9446ea3-dac9-4a1a-9ad3-1fda85d9f151-webhook-cert\") pod \"opendatahub-operator-controller-manager-5cf8c55d45-8d6hh\" (UID: \"c9446ea3-dac9-4a1a-9ad3-1fda85d9f151\") " pod="opendatahub/opendatahub-operator-controller-manager-5cf8c55d45-8d6hh"
Jul 16 00:46:09.853820 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:46:09.853794    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c9446ea3-dac9-4a1a-9ad3-1fda85d9f151-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5cf8c55d45-8d6hh\" (UID: \"c9446ea3-dac9-4a1a-9ad3-1fda85d9f151\") " pod="opendatahub/opendatahub-operator-controller-manager-5cf8c55d45-8d6hh"
Jul 16 00:46:09.858905 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:46:09.858878    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l69ts\" (UniqueName: \"kubernetes.io/projected/c9446ea3-dac9-4a1a-9ad3-1fda85d9f151-kube-api-access-l69ts\") pod \"opendatahub-operator-controller-manager-5cf8c55d45-8d6hh\" (UID: \"c9446ea3-dac9-4a1a-9ad3-1fda85d9f151\") " pod="opendatahub/opendatahub-operator-controller-manager-5cf8c55d45-8d6hh"
Jul 16 00:46:09.907980 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:46:09.907943    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5cf8c55d45-8d6hh"
Jul 16 00:46:10.047070 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:46:10.047045    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5cf8c55d45-8d6hh"]
Jul 16 00:46:10.049825 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:46:10.049793    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9446ea3_dac9_4a1a_9ad3_1fda85d9f151.slice/crio-abd12700878e949d03173ae7565e57f31ee121fd8720a34440149b41e15e04b9 WatchSource:0}: Error finding container abd12700878e949d03173ae7565e57f31ee121fd8720a34440149b41e15e04b9: Status 404 returned error can't find the container with id abd12700878e949d03173ae7565e57f31ee121fd8720a34440149b41e15e04b9
Jul 16 00:46:10.458002 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:46:10.457964    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5cf8c55d45-8d6hh" event={"ID":"c9446ea3-dac9-4a1a-9ad3-1fda85d9f151","Type":"ContainerStarted","Data":"abd12700878e949d03173ae7565e57f31ee121fd8720a34440149b41e15e04b9"}
Jul 16 00:46:13.475453 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:46:13.475365    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5cf8c55d45-8d6hh" event={"ID":"c9446ea3-dac9-4a1a-9ad3-1fda85d9f151","Type":"ContainerStarted","Data":"b6d8b5d451fe9b2b3b3fb37aa79f347ca2dab3cbe06f4bc278cca5edeb03eb7f"}
Jul 16 00:46:14.481552 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:46:14.481520    2563 generic.go:356] "Generic (PLEG): container finished" podID="c9446ea3-dac9-4a1a-9ad3-1fda85d9f151" containerID="b6d8b5d451fe9b2b3b3fb37aa79f347ca2dab3cbe06f4bc278cca5edeb03eb7f" exitCode=0
Jul 16 00:46:14.481995 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:46:14.481589    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5cf8c55d45-8d6hh" event={"ID":"c9446ea3-dac9-4a1a-9ad3-1fda85d9f151","Type":"ContainerDied","Data":"b6d8b5d451fe9b2b3b3fb37aa79f347ca2dab3cbe06f4bc278cca5edeb03eb7f"}
Jul 16 00:46:15.440428 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:46:15.440395    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-c857fdb5b-5bxrx"
Jul 16 00:46:15.488839 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:46:15.488801    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5cf8c55d45-8d6hh" event={"ID":"c9446ea3-dac9-4a1a-9ad3-1fda85d9f151","Type":"ContainerStarted","Data":"e020ad4190bf4be37cf871fec63d9e1c022877511582cd2c013c368178e965d5"}
Jul 16 00:46:15.489321 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:46:15.488977    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-5cf8c55d45-8d6hh"
Jul 16 00:46:15.521578 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:46:15.521490    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-5cf8c55d45-8d6hh" podStartSLOduration=1.9367652199999998 podStartE2EDuration="6.521465362s" totalImagesPullingTime="3.537743481s" totalInitContainerRuntime="1.046956661s" isStatefulPod=false podCreationTimestamp="2026-07-16 00:46:09 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 00:46:15.518753611 +0000 UTC m=+347.954015351" watchObservedRunningTime="2026-07-16 00:46:15.521465362 +0000 UTC m=+347.956727102"
Jul 16 00:46:26.495557 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:46:26.495471    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-5cf8c55d45-8d6hh"
Jul 16 00:46:29.861980 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:46:29.861940    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7ffc9f7c5d-k7f7b"]
Jul 16 00:46:29.873993 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:46:29.873967    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7ffc9f7c5d-k7f7b"
Jul 16 00:46:29.877900 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:46:29.877856    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 16 00:46:29.879035 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:46:29.879016    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 16 00:46:29.879587 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:46:29.879569    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Jul 16 00:46:29.879903 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:46:29.879884    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 16 00:46:29.880629 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:46:29.880608    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-k8cvz\""
Jul 16 00:46:29.908211 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:46:29.908181    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b98801ba-36e7-4e48-84b8-56709b911b43-tmp\") pod \"kube-auth-proxy-7ffc9f7c5d-k7f7b\" (UID: \"b98801ba-36e7-4e48-84b8-56709b911b43\") " pod="openshift-ingress/kube-auth-proxy-7ffc9f7c5d-k7f7b"
Jul 16 00:46:29.908361 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:46:29.908222    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tj4z\" (UniqueName: \"kubernetes.io/projected/b98801ba-36e7-4e48-84b8-56709b911b43-kube-api-access-5tj4z\") pod \"kube-auth-proxy-7ffc9f7c5d-k7f7b\" (UID: \"b98801ba-36e7-4e48-84b8-56709b911b43\") " pod="openshift-ingress/kube-auth-proxy-7ffc9f7c5d-k7f7b"
Jul 16 00:46:29.908361 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:46:29.908257    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b98801ba-36e7-4e48-84b8-56709b911b43-tls-certs\") pod \"kube-auth-proxy-7ffc9f7c5d-k7f7b\" (UID: \"b98801ba-36e7-4e48-84b8-56709b911b43\") " pod="openshift-ingress/kube-auth-proxy-7ffc9f7c5d-k7f7b"
Jul 16 00:46:29.912160 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:46:29.912131    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7ffc9f7c5d-k7f7b"]
Jul 16 00:46:30.009282 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:46:30.009254    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b98801ba-36e7-4e48-84b8-56709b911b43-tmp\") pod \"kube-auth-proxy-7ffc9f7c5d-k7f7b\" (UID: \"b98801ba-36e7-4e48-84b8-56709b911b43\") " pod="openshift-ingress/kube-auth-proxy-7ffc9f7c5d-k7f7b"
Jul 16 00:46:30.009282 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:46:30.009286    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tj4z\" (UniqueName: \"kubernetes.io/projected/b98801ba-36e7-4e48-84b8-56709b911b43-kube-api-access-5tj4z\") pod \"kube-auth-proxy-7ffc9f7c5d-k7f7b\" (UID: \"b98801ba-36e7-4e48-84b8-56709b911b43\") " pod="openshift-ingress/kube-auth-proxy-7ffc9f7c5d-k7f7b"
Jul 16 00:46:30.009479 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:46:30.009313    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b98801ba-36e7-4e48-84b8-56709b911b43-tls-certs\") pod \"kube-auth-proxy-7ffc9f7c5d-k7f7b\" (UID: \"b98801ba-36e7-4e48-84b8-56709b911b43\") " pod="openshift-ingress/kube-auth-proxy-7ffc9f7c5d-k7f7b"
Jul 16 00:46:30.009479 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:46:30.009464    2563 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found
Jul 16 00:46:30.009548 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:46:30.009525    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b98801ba-36e7-4e48-84b8-56709b911b43-tls-certs podName:b98801ba-36e7-4e48-84b8-56709b911b43 nodeName:}" failed. No retries permitted until 2026-07-16 00:46:30.509503659 +0000 UTC m=+362.944765392 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/b98801ba-36e7-4e48-84b8-56709b911b43-tls-certs") pod "kube-auth-proxy-7ffc9f7c5d-k7f7b" (UID: "b98801ba-36e7-4e48-84b8-56709b911b43") : secret "kube-auth-proxy-tls" not found
Jul 16 00:46:30.011542 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:46:30.011513    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b98801ba-36e7-4e48-84b8-56709b911b43-tmp\") pod \"kube-auth-proxy-7ffc9f7c5d-k7f7b\" (UID: \"b98801ba-36e7-4e48-84b8-56709b911b43\") " pod="openshift-ingress/kube-auth-proxy-7ffc9f7c5d-k7f7b"
Jul 16 00:46:30.030789 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:46:30.030761    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tj4z\" (UniqueName: \"kubernetes.io/projected/b98801ba-36e7-4e48-84b8-56709b911b43-kube-api-access-5tj4z\") pod \"kube-auth-proxy-7ffc9f7c5d-k7f7b\" (UID: \"b98801ba-36e7-4e48-84b8-56709b911b43\") " pod="openshift-ingress/kube-auth-proxy-7ffc9f7c5d-k7f7b"
Jul 16 00:46:30.513983 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:46:30.513949    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b98801ba-36e7-4e48-84b8-56709b911b43-tls-certs\") pod \"kube-auth-proxy-7ffc9f7c5d-k7f7b\" (UID: \"b98801ba-36e7-4e48-84b8-56709b911b43\") " pod="openshift-ingress/kube-auth-proxy-7ffc9f7c5d-k7f7b"
Jul 16 00:46:30.516514 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:46:30.516485    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b98801ba-36e7-4e48-84b8-56709b911b43-tls-certs\") pod \"kube-auth-proxy-7ffc9f7c5d-k7f7b\" (UID: \"b98801ba-36e7-4e48-84b8-56709b911b43\") " pod="openshift-ingress/kube-auth-proxy-7ffc9f7c5d-k7f7b"
Jul 16 00:46:30.783493 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:46:30.783410    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7ffc9f7c5d-k7f7b"
Jul 16 00:46:30.970206 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:46:30.970173    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7ffc9f7c5d-k7f7b"]
Jul 16 00:46:30.973221 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:46:30.973191    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb98801ba_36e7_4e48_84b8_56709b911b43.slice/crio-b979cc948ebc666a66c86080ddc3d0b40d8d34922275f7c59e7cb6d114bdcaa6 WatchSource:0}: Error finding container b979cc948ebc666a66c86080ddc3d0b40d8d34922275f7c59e7cb6d114bdcaa6: Status 404 returned error can't find the container with id b979cc948ebc666a66c86080ddc3d0b40d8d34922275f7c59e7cb6d114bdcaa6
Jul 16 00:46:31.555644 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:46:31.555598    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7ffc9f7c5d-k7f7b" event={"ID":"b98801ba-36e7-4e48-84b8-56709b911b43","Type":"ContainerStarted","Data":"b979cc948ebc666a66c86080ddc3d0b40d8d34922275f7c59e7cb6d114bdcaa6"}
Jul 16 00:46:34.568273 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:46:34.568232    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7ffc9f7c5d-k7f7b" event={"ID":"b98801ba-36e7-4e48-84b8-56709b911b43","Type":"ContainerStarted","Data":"7c4308b11d2b8248d1cdcf43899b79dfa33976225f0f36a70b565c5da0589db0"}
Jul 16 00:46:34.593242 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:46:34.593184    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7ffc9f7c5d-k7f7b" podStartSLOduration=3.026590084 podStartE2EDuration="5.593165816s" totalImagesPullingTime="2.566575732s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 00:46:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 00:46:34.591003969 +0000 UTC m=+367.026265719" watchObservedRunningTime="2026-07-16 00:46:34.593165816 +0000 UTC m=+367.028427556"
Jul 16 00:46:43.219319 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:46:43.219286    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-cgjd4"]
Jul 16 00:46:43.222937 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:46:43.222917    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-cgjd4"
Jul 16 00:46:43.230349 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:46:43.230329    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-6xljn\""
Jul 16 00:46:43.231159 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:46:43.231143    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Jul 16 00:46:43.264195 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:46:43.264166    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-cgjd4"]
Jul 16 00:46:43.338603 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:46:43.338566    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f670d048-05a1-44e0-bb47-f33afeba3880-cert\") pod \"kserve-controller-manager-d6f4b65b8-cgjd4\" (UID: \"f670d048-05a1-44e0-bb47-f33afeba3880\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-cgjd4"
Jul 16 00:46:43.338770 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:46:43.338609    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48jws\" (UniqueName: \"kubernetes.io/projected/f670d048-05a1-44e0-bb47-f33afeba3880-kube-api-access-48jws\") pod \"kserve-controller-manager-d6f4b65b8-cgjd4\" (UID: \"f670d048-05a1-44e0-bb47-f33afeba3880\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-cgjd4"
Jul 16 00:46:43.439970 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:46:43.439930    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f670d048-05a1-44e0-bb47-f33afeba3880-cert\") pod \"kserve-controller-manager-d6f4b65b8-cgjd4\" (UID: \"f670d048-05a1-44e0-bb47-f33afeba3880\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-cgjd4"
Jul 16 00:46:43.440163 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:46:43.439979    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48jws\" (UniqueName: \"kubernetes.io/projected/f670d048-05a1-44e0-bb47-f33afeba3880-kube-api-access-48jws\") pod \"kserve-controller-manager-d6f4b65b8-cgjd4\" (UID: \"f670d048-05a1-44e0-bb47-f33afeba3880\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-cgjd4"
Jul 16 00:46:43.442376 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:46:43.442355    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f670d048-05a1-44e0-bb47-f33afeba3880-cert\") pod \"kserve-controller-manager-d6f4b65b8-cgjd4\" (UID: \"f670d048-05a1-44e0-bb47-f33afeba3880\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-cgjd4"
Jul 16 00:46:43.467793 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:46:43.467759    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-48jws\" (UniqueName: \"kubernetes.io/projected/f670d048-05a1-44e0-bb47-f33afeba3880-kube-api-access-48jws\") pod \"kserve-controller-manager-d6f4b65b8-cgjd4\" (UID: \"f670d048-05a1-44e0-bb47-f33afeba3880\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-cgjd4"
Jul 16 00:46:43.533438 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:46:43.533351    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-cgjd4"
Jul 16 00:46:43.710495 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:46:43.710374    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-cgjd4"]
Jul 16 00:46:43.713182 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:46:43.713147    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf670d048_05a1_44e0_bb47_f33afeba3880.slice/crio-44dc750466804a4d20097176d01076b6d220deaff388bde21738394b06c3bb52 WatchSource:0}: Error finding container 44dc750466804a4d20097176d01076b6d220deaff388bde21738394b06c3bb52: Status 404 returned error can't find the container with id 44dc750466804a4d20097176d01076b6d220deaff388bde21738394b06c3bb52
Jul 16 00:46:44.620911 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:46:44.620875    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-cgjd4" event={"ID":"f670d048-05a1-44e0-bb47-f33afeba3880","Type":"ContainerStarted","Data":"44dc750466804a4d20097176d01076b6d220deaff388bde21738394b06c3bb52"}
Jul 16 00:46:46.632454 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:46:46.632372    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-cgjd4" event={"ID":"f670d048-05a1-44e0-bb47-f33afeba3880","Type":"ContainerStarted","Data":"657715f3e700152f86167e6fe740464f5e1490fb1e6cdb85d47b0e5329cc7daf"}
Jul 16 00:46:46.632454 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:46:46.632444    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-cgjd4"
Jul 16 00:46:46.668706 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:46:46.668656    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-cgjd4" podStartSLOduration=1.036415893 podStartE2EDuration="3.668642028s" totalImagesPullingTime="2.632226135s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 00:46:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 00:46:46.667131859 +0000 UTC m=+379.102393599" watchObservedRunningTime="2026-07-16 00:46:46.668642028 +0000 UTC m=+379.103903766"
Jul 16 00:47:17.642419 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:47:17.642387    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-cgjd4"
Jul 16 00:48:17.440376 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:48:17.440297    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-pv7nf"]
Jul 16 00:48:17.443726 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:48:17.443709    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-pv7nf"
Jul 16 00:48:17.450114 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:48:17.445288    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5582\" (UniqueName: \"kubernetes.io/projected/36c1deac-8dd8-4797-a13f-f335d7ddcb12-kube-api-access-m5582\") pod \"kuadrant-console-plugin-66b5774985-pv7nf\" (UID: \"36c1deac-8dd8-4797-a13f-f335d7ddcb12\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-pv7nf"
Jul 16 00:48:17.450114 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:48:17.445354    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/36c1deac-8dd8-4797-a13f-f335d7ddcb12-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-pv7nf\" (UID: \"36c1deac-8dd8-4797-a13f-f335d7ddcb12\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-pv7nf"
Jul 16 00:48:17.450114 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:48:17.445422    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/36c1deac-8dd8-4797-a13f-f335d7ddcb12-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-pv7nf\" (UID: \"36c1deac-8dd8-4797-a13f-f335d7ddcb12\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-pv7nf"
Jul 16 00:48:17.455527 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:48:17.455502    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Jul 16 00:48:17.455766 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:48:17.455522    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 16 00:48:17.455882 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:48:17.455564    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 16 00:48:17.455882 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:48:17.455589    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Jul 16 00:48:17.456040 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:48:17.455996    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-tcss8\""
Jul 16 00:48:17.465663 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:48:17.465644    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-pv7nf"]
Jul 16 00:48:17.545845 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:48:17.545807    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/36c1deac-8dd8-4797-a13f-f335d7ddcb12-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-pv7nf\" (UID: \"36c1deac-8dd8-4797-a13f-f335d7ddcb12\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-pv7nf"
Jul 16 00:48:17.546021 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:48:17.545882    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/36c1deac-8dd8-4797-a13f-f335d7ddcb12-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-pv7nf\" (UID: \"36c1deac-8dd8-4797-a13f-f335d7ddcb12\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-pv7nf"
Jul 16 00:48:17.546021 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:48:17.545940    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5582\" (UniqueName: \"kubernetes.io/projected/36c1deac-8dd8-4797-a13f-f335d7ddcb12-kube-api-access-m5582\") pod \"kuadrant-console-plugin-66b5774985-pv7nf\" (UID: \"36c1deac-8dd8-4797-a13f-f335d7ddcb12\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-pv7nf"
Jul 16 00:48:17.546493 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:48:17.546472    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/36c1deac-8dd8-4797-a13f-f335d7ddcb12-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-pv7nf\" (UID: \"36c1deac-8dd8-4797-a13f-f335d7ddcb12\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-pv7nf"
Jul 16 00:48:17.548392 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:48:17.548369    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/36c1deac-8dd8-4797-a13f-f335d7ddcb12-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-pv7nf\" (UID: \"36c1deac-8dd8-4797-a13f-f335d7ddcb12\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-pv7nf"
Jul 16 00:48:17.559672 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:48:17.559643    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5582\" (UniqueName: \"kubernetes.io/projected/36c1deac-8dd8-4797-a13f-f335d7ddcb12-kube-api-access-m5582\") pod \"kuadrant-console-plugin-66b5774985-pv7nf\" (UID: \"36c1deac-8dd8-4797-a13f-f335d7ddcb12\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-pv7nf"
Jul 16 00:48:17.759445 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:48:17.759361    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-pv7nf"
Jul 16 00:48:17.892179 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:48:17.892080    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-pv7nf"]
Jul 16 00:48:17.894472 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:48:17.894442    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod36c1deac_8dd8_4797_a13f_f335d7ddcb12.slice/crio-8ca5fea1e2e4fd557232082654465100f6eee2b15ea357d78683dfea69f977ef WatchSource:0}: Error finding container 8ca5fea1e2e4fd557232082654465100f6eee2b15ea357d78683dfea69f977ef: Status 404 returned error can't find the container with id 8ca5fea1e2e4fd557232082654465100f6eee2b15ea357d78683dfea69f977ef
Jul 16 00:48:18.028823 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:48:18.028728    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-pv7nf" event={"ID":"36c1deac-8dd8-4797-a13f-f335d7ddcb12","Type":"ContainerStarted","Data":"8ca5fea1e2e4fd557232082654465100f6eee2b15ea357d78683dfea69f977ef"}
Jul 16 00:48:40.143584 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:48:40.143536    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-pv7nf" event={"ID":"36c1deac-8dd8-4797-a13f-f335d7ddcb12","Type":"ContainerStarted","Data":"e82d807c5fd77c2d0e39a016fb019ed8b6e1dd8b5c8863269ce25f230cab871e"}
Jul 16 00:48:40.159026 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:48:40.158978    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-pv7nf" podStartSLOduration=1.5339661 podStartE2EDuration="23.158963192s" totalImagesPullingTime="21.624997092s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 00:48:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 00:48:40.157672687 +0000 UTC m=+492.592934425" watchObservedRunningTime="2026-07-16 00:48:40.158963192 +0000 UTC m=+492.594224930"
Jul 16 00:48:59.550414 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:48:59.550377    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl7fx"]
Jul 16 00:48:59.553667 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:48:59.553644    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-hl7fx"
Jul 16 00:48:59.555601 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:48:59.555575    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 16 00:48:59.562243 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:48:59.562221    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl7fx"]
Jul 16 00:48:59.594631 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:48:59.594597    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl7fx"]
Jul 16 00:48:59.644353 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:48:59.644320    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjj7h\" (UniqueName: \"kubernetes.io/projected/991a6621-30fa-4462-8b75-7acfbd97cb4a-kube-api-access-vjj7h\") pod \"limitador-limitador-54d6d988d5-hl7fx\" (UID: \"991a6621-30fa-4462-8b75-7acfbd97cb4a\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-hl7fx"
Jul 16 00:48:59.644524 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:48:59.644384    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/991a6621-30fa-4462-8b75-7acfbd97cb4a-config-file\") pod \"limitador-limitador-54d6d988d5-hl7fx\" (UID: \"991a6621-30fa-4462-8b75-7acfbd97cb4a\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-hl7fx"
Jul 16 00:48:59.745721 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:48:59.745688    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/991a6621-30fa-4462-8b75-7acfbd97cb4a-config-file\") pod \"limitador-limitador-54d6d988d5-hl7fx\" (UID: \"991a6621-30fa-4462-8b75-7acfbd97cb4a\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-hl7fx"
Jul 16 00:48:59.745915 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:48:59.745783    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjj7h\" (UniqueName: \"kubernetes.io/projected/991a6621-30fa-4462-8b75-7acfbd97cb4a-kube-api-access-vjj7h\") pod \"limitador-limitador-54d6d988d5-hl7fx\" (UID: \"991a6621-30fa-4462-8b75-7acfbd97cb4a\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-hl7fx"
Jul 16 00:48:59.746413 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:48:59.746393    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/991a6621-30fa-4462-8b75-7acfbd97cb4a-config-file\") pod \"limitador-limitador-54d6d988d5-hl7fx\" (UID: \"991a6621-30fa-4462-8b75-7acfbd97cb4a\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-hl7fx"
Jul 16 00:48:59.754596 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:48:59.754576    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjj7h\" (UniqueName: \"kubernetes.io/projected/991a6621-30fa-4462-8b75-7acfbd97cb4a-kube-api-access-vjj7h\") pod \"limitador-limitador-54d6d988d5-hl7fx\" (UID: \"991a6621-30fa-4462-8b75-7acfbd97cb4a\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-hl7fx"
Jul 16 00:48:59.863717 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:48:59.863678    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-hl7fx"
Jul 16 00:49:00.006336 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:49:00.006309    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl7fx"]
Jul 16 00:49:00.009100 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:49:00.009075    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod991a6621_30fa_4462_8b75_7acfbd97cb4a.slice/crio-e86626d76134d21a4555e6666919a22b3b4f79d598483a65503041a822090389 WatchSource:0}: Error finding container e86626d76134d21a4555e6666919a22b3b4f79d598483a65503041a822090389: Status 404 returned error can't find the container with id e86626d76134d21a4555e6666919a22b3b4f79d598483a65503041a822090389
Jul 16 00:49:00.028673 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:49:00.028640    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-xcbpc"]
Jul 16 00:49:00.033462 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:49:00.033443    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-xcbpc"
Jul 16 00:49:00.036504 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:49:00.036485    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-7cmff\""
Jul 16 00:49:00.042250 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:49:00.042227    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-xcbpc"]
Jul 16 00:49:00.149560 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:49:00.149468    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2ntr\" (UniqueName: \"kubernetes.io/projected/90da2360-bfb7-49d3-97f4-198bb47efc7f-kube-api-access-h2ntr\") pod \"authorino-58cd955675-xcbpc\" (UID: \"90da2360-bfb7-49d3-97f4-198bb47efc7f\") " pod="kuadrant-system/authorino-58cd955675-xcbpc"
Jul 16 00:49:00.234360 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:49:00.234330    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-hl7fx" event={"ID":"991a6621-30fa-4462-8b75-7acfbd97cb4a","Type":"ContainerStarted","Data":"e86626d76134d21a4555e6666919a22b3b4f79d598483a65503041a822090389"}
Jul 16 00:49:00.249998 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:49:00.249969    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2ntr\" (UniqueName: \"kubernetes.io/projected/90da2360-bfb7-49d3-97f4-198bb47efc7f-kube-api-access-h2ntr\") pod \"authorino-58cd955675-xcbpc\" (UID: \"90da2360-bfb7-49d3-97f4-198bb47efc7f\") " pod="kuadrant-system/authorino-58cd955675-xcbpc"
Jul 16 00:49:00.261392 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:49:00.261361    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2ntr\" (UniqueName: \"kubernetes.io/projected/90da2360-bfb7-49d3-97f4-198bb47efc7f-kube-api-access-h2ntr\") pod \"authorino-58cd955675-xcbpc\" (UID: \"90da2360-bfb7-49d3-97f4-198bb47efc7f\") " pod="kuadrant-system/authorino-58cd955675-xcbpc"
Jul 16 00:49:00.343021 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:49:00.342987    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-xcbpc"
Jul 16 00:49:00.474808 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:49:00.474781    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-xcbpc"]
Jul 16 00:49:00.477409 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:49:00.477367    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod90da2360_bfb7_49d3_97f4_198bb47efc7f.slice/crio-bc86c54cd215a51e90878e4e7f3d067fd28f8f58ef50d966b53a1c789209599b WatchSource:0}: Error finding container bc86c54cd215a51e90878e4e7f3d067fd28f8f58ef50d966b53a1c789209599b: Status 404 returned error can't find the container with id bc86c54cd215a51e90878e4e7f3d067fd28f8f58ef50d966b53a1c789209599b
Jul 16 00:49:01.244979 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:49:01.244905    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-xcbpc" event={"ID":"90da2360-bfb7-49d3-97f4-198bb47efc7f","Type":"ContainerStarted","Data":"bc86c54cd215a51e90878e4e7f3d067fd28f8f58ef50d966b53a1c789209599b"}
Jul 16 00:49:04.070292 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:49:04.070246    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-xcbpc"]
Jul 16 00:49:05.267343 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:49:05.267299    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-xcbpc" event={"ID":"90da2360-bfb7-49d3-97f4-198bb47efc7f","Type":"ContainerStarted","Data":"1e67fac98cf17fe6c1b2101a538c59e3e7c5a38374c32b8cfff589b4fe5ae726"}
Jul 16 00:49:05.267951 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:49:05.267366    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-xcbpc" podUID="90da2360-bfb7-49d3-97f4-198bb47efc7f" containerName="authorino" containerID="cri-o://1e67fac98cf17fe6c1b2101a538c59e3e7c5a38374c32b8cfff589b4fe5ae726" gracePeriod=30
Jul 16 00:49:05.268901 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:49:05.268856    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-hl7fx" event={"ID":"991a6621-30fa-4462-8b75-7acfbd97cb4a","Type":"ContainerStarted","Data":"a15c84db4f92d7d768fadbc2bbe5a7eaad969c26057e6903e7e3d9e92650ae34"}
Jul 16 00:49:05.269001 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:49:05.268963    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-hl7fx"
Jul 16 00:49:05.282971 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:49:05.282926    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-xcbpc" podStartSLOduration=1.215099492 podStartE2EDuration="5.282909138s" totalImagesPullingTime="4.067809646s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 00:49:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 00:49:05.281037082 +0000 UTC m=+517.716298832" watchObservedRunningTime="2026-07-16 00:49:05.282909138 +0000 UTC m=+517.718170875"
Jul 16 00:49:05.296933 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:49:05.296858    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-hl7fx" podStartSLOduration=1.76875626 podStartE2EDuration="6.296843684s" totalImagesPullingTime="4.528087424s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 00:48:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 00:49:05.295964638 +0000 UTC m=+517.731226378" watchObservedRunningTime="2026-07-16 00:49:05.296843684 +0000 UTC m=+517.732105423"
Jul 16 00:49:05.517121 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:49:05.517095    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-xcbpc"
Jul 16 00:49:05.595182 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:49:05.595146    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h2ntr\" (UniqueName: \"kubernetes.io/projected/90da2360-bfb7-49d3-97f4-198bb47efc7f-kube-api-access-h2ntr\") pod \"90da2360-bfb7-49d3-97f4-198bb47efc7f\" (UID: \"90da2360-bfb7-49d3-97f4-198bb47efc7f\") "
Jul 16 00:49:05.597278 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:49:05.597246    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90da2360-bfb7-49d3-97f4-198bb47efc7f-kube-api-access-h2ntr" (OuterVolumeSpecName: "kube-api-access-h2ntr") pod "90da2360-bfb7-49d3-97f4-198bb47efc7f" (UID: "90da2360-bfb7-49d3-97f4-198bb47efc7f"). InnerVolumeSpecName "kube-api-access-h2ntr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 00:49:05.696369 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:49:05.696330    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2ntr\" (UniqueName: \"kubernetes.io/projected/90da2360-bfb7-49d3-97f4-198bb47efc7f-kube-api-access-h2ntr\") on node \"ip-10-0-143-65.ec2.internal\" DevicePath \"\""
Jul 16 00:49:06.274411 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:49:06.274329    2563 generic.go:356] "Generic (PLEG): container finished" podID="90da2360-bfb7-49d3-97f4-198bb47efc7f" containerID="1e67fac98cf17fe6c1b2101a538c59e3e7c5a38374c32b8cfff589b4fe5ae726" exitCode=0
Jul 16 00:49:06.274411 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:49:06.274368    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-xcbpc"
Jul 16 00:49:06.274411 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:49:06.274376    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-xcbpc" event={"ID":"90da2360-bfb7-49d3-97f4-198bb47efc7f","Type":"ContainerDied","Data":"1e67fac98cf17fe6c1b2101a538c59e3e7c5a38374c32b8cfff589b4fe5ae726"}
Jul 16 00:49:06.274903 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:49:06.274422    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-xcbpc" event={"ID":"90da2360-bfb7-49d3-97f4-198bb47efc7f","Type":"ContainerDied","Data":"bc86c54cd215a51e90878e4e7f3d067fd28f8f58ef50d966b53a1c789209599b"}
Jul 16 00:49:06.274903 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:49:06.274445    2563 scope.go:117] "RemoveContainer" containerID="1e67fac98cf17fe6c1b2101a538c59e3e7c5a38374c32b8cfff589b4fe5ae726"
Jul 16 00:49:06.282679 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:49:06.282661    2563 scope.go:117] "RemoveContainer" containerID="1e67fac98cf17fe6c1b2101a538c59e3e7c5a38374c32b8cfff589b4fe5ae726"
Jul 16 00:49:06.282940 ip-10-0-143-65 kubenswrapper[2563]: E0716 00:49:06.282922    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e67fac98cf17fe6c1b2101a538c59e3e7c5a38374c32b8cfff589b4fe5ae726\": container with ID starting with 1e67fac98cf17fe6c1b2101a538c59e3e7c5a38374c32b8cfff589b4fe5ae726 not found: ID does not exist" containerID="1e67fac98cf17fe6c1b2101a538c59e3e7c5a38374c32b8cfff589b4fe5ae726"
Jul 16 00:49:06.282992 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:49:06.282949    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e67fac98cf17fe6c1b2101a538c59e3e7c5a38374c32b8cfff589b4fe5ae726"} err="failed to get container status \"1e67fac98cf17fe6c1b2101a538c59e3e7c5a38374c32b8cfff589b4fe5ae726\": rpc error: code = NotFound desc = could not find container \"1e67fac98cf17fe6c1b2101a538c59e3e7c5a38374c32b8cfff589b4fe5ae726\": container with ID starting with 1e67fac98cf17fe6c1b2101a538c59e3e7c5a38374c32b8cfff589b4fe5ae726 not found: ID does not exist"
Jul 16 00:49:06.294425 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:49:06.294399    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-xcbpc"]
Jul 16 00:49:06.298464 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:49:06.298432    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-xcbpc"]
Jul 16 00:49:08.005542 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:49:08.005506    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="90da2360-bfb7-49d3-97f4-198bb47efc7f" path="/var/lib/kubelet/pods/90da2360-bfb7-49d3-97f4-198bb47efc7f/volumes"
Jul 16 00:49:16.276031 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:49:16.276000    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-hl7fx"
Jul 16 00:50:15.173019 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:50:15.172983    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl7fx"]
Jul 16 00:50:27.886827 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:50:27.886801    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bn5hr_e990a21d-614b-4d1d-be54-0df6e44487ed/ovn-acl-logging/0.log"
Jul 16 00:50:27.887476 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:50:27.887456    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bn5hr_e990a21d-614b-4d1d-be54-0df6e44487ed/ovn-acl-logging/0.log"
Jul 16 00:50:42.741679 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:50:42.741645    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl7fx"]
Jul 16 00:50:52.244331 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:50:52.244251    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl7fx"]
Jul 16 00:51:11.247162 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:51:11.247121    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl7fx"]
Jul 16 00:51:17.068212 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:51:17.068178    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl7fx"]
Jul 16 00:51:22.535792 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:51:22.535747    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-z6sx5"]
Jul 16 00:51:22.536214 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:51:22.536125    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90da2360-bfb7-49d3-97f4-198bb47efc7f" containerName="authorino"
Jul 16 00:51:22.536214 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:51:22.536144    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="90da2360-bfb7-49d3-97f4-198bb47efc7f" containerName="authorino"
Jul 16 00:51:22.536296 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:51:22.536239    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="90da2360-bfb7-49d3-97f4-198bb47efc7f" containerName="authorino"
Jul 16 00:51:22.539116 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:51:22.539096    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-z6sx5"
Jul 16 00:51:22.541212 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:51:22.541185    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 16 00:51:22.541354 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:51:22.541332    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 16 00:51:22.541786 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:51:22.541767    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Jul 16 00:51:22.541908 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:51:22.541792    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-scpnw\""
Jul 16 00:51:22.549615 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:51:22.549589    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-z6sx5"]
Jul 16 00:51:22.683339 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:51:22.683299    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2d0e466-a859-4e50-8703-43adcdcb1aaa-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-z6sx5\" (UID: \"b2d0e466-a859-4e50-8703-43adcdcb1aaa\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-z6sx5"
Jul 16 00:51:22.683538 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:51:22.683374    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2d0e466-a859-4e50-8703-43adcdcb1aaa-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-z6sx5\" (UID: \"b2d0e466-a859-4e50-8703-43adcdcb1aaa\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-z6sx5"
Jul 16 00:51:22.683538 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:51:22.683396    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2d0e466-a859-4e50-8703-43adcdcb1aaa-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-z6sx5\" (UID: \"b2d0e466-a859-4e50-8703-43adcdcb1aaa\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-z6sx5"
Jul 16 00:51:22.683538 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:51:22.683417    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2d0e466-a859-4e50-8703-43adcdcb1aaa-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-z6sx5\" (UID: \"b2d0e466-a859-4e50-8703-43adcdcb1aaa\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-z6sx5"
Jul 16 00:51:22.683538 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:51:22.683434    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rsfc\" (UniqueName: \"kubernetes.io/projected/b2d0e466-a859-4e50-8703-43adcdcb1aaa-kube-api-access-9rsfc\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-z6sx5\" (UID: \"b2d0e466-a859-4e50-8703-43adcdcb1aaa\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-z6sx5"
Jul 16 00:51:22.683713 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:51:22.683536    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2d0e466-a859-4e50-8703-43adcdcb1aaa-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-z6sx5\" (UID: \"b2d0e466-a859-4e50-8703-43adcdcb1aaa\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-z6sx5"
Jul 16 00:51:22.784495 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:51:22.784449    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2d0e466-a859-4e50-8703-43adcdcb1aaa-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-z6sx5\" (UID: \"b2d0e466-a859-4e50-8703-43adcdcb1aaa\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-z6sx5"
Jul 16 00:51:22.784701 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:51:22.784509    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2d0e466-a859-4e50-8703-43adcdcb1aaa-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-z6sx5\" (UID: \"b2d0e466-a859-4e50-8703-43adcdcb1aaa\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-z6sx5"
Jul 16 00:51:22.784701 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:51:22.784574    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2d0e466-a859-4e50-8703-43adcdcb1aaa-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-z6sx5\" (UID: \"b2d0e466-a859-4e50-8703-43adcdcb1aaa\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-z6sx5"
Jul 16 00:51:22.784701 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:51:22.784593    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2d0e466-a859-4e50-8703-43adcdcb1aaa-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-z6sx5\" (UID: \"b2d0e466-a859-4e50-8703-43adcdcb1aaa\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-z6sx5"
Jul 16 00:51:22.784701 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:51:22.784615    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2d0e466-a859-4e50-8703-43adcdcb1aaa-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-z6sx5\" (UID: \"b2d0e466-a859-4e50-8703-43adcdcb1aaa\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-z6sx5"
Jul 16 00:51:22.784701 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:51:22.784633    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rsfc\" (UniqueName: \"kubernetes.io/projected/b2d0e466-a859-4e50-8703-43adcdcb1aaa-kube-api-access-9rsfc\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-z6sx5\" (UID: \"b2d0e466-a859-4e50-8703-43adcdcb1aaa\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-z6sx5"
Jul 16 00:51:22.785003 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:51:22.784957    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2d0e466-a859-4e50-8703-43adcdcb1aaa-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-z6sx5\" (UID: \"b2d0e466-a859-4e50-8703-43adcdcb1aaa\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-z6sx5"
Jul 16 00:51:22.785003 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:51:22.784971    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2d0e466-a859-4e50-8703-43adcdcb1aaa-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-z6sx5\" (UID: \"b2d0e466-a859-4e50-8703-43adcdcb1aaa\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-z6sx5"
Jul 16 00:51:22.785073 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:51:22.785024    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2d0e466-a859-4e50-8703-43adcdcb1aaa-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-z6sx5\" (UID: \"b2d0e466-a859-4e50-8703-43adcdcb1aaa\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-z6sx5"
Jul 16 00:51:22.787042 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:51:22.786985    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2d0e466-a859-4e50-8703-43adcdcb1aaa-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-z6sx5\" (UID: \"b2d0e466-a859-4e50-8703-43adcdcb1aaa\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-z6sx5"
Jul 16 00:51:22.787147 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:51:22.787130    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2d0e466-a859-4e50-8703-43adcdcb1aaa-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-z6sx5\" (UID: \"b2d0e466-a859-4e50-8703-43adcdcb1aaa\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-z6sx5"
Jul 16 00:51:22.792187 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:51:22.792163    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rsfc\" (UniqueName: \"kubernetes.io/projected/b2d0e466-a859-4e50-8703-43adcdcb1aaa-kube-api-access-9rsfc\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-z6sx5\" (UID: \"b2d0e466-a859-4e50-8703-43adcdcb1aaa\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-z6sx5"
Jul 16 00:51:22.849932 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:51:22.849885    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-z6sx5"
Jul 16 00:51:22.996786 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:51:22.996759    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-z6sx5"]
Jul 16 00:51:22.999162 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:51:22.999135    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2d0e466_a859_4e50_8703_43adcdcb1aaa.slice/crio-9df94d319888785c7b89b1c2ad472df8e943e07bb36a2e4511b45996f8d4858a WatchSource:0}: Error finding container 9df94d319888785c7b89b1c2ad472df8e943e07bb36a2e4511b45996f8d4858a: Status 404 returned error can't find the container with id 9df94d319888785c7b89b1c2ad472df8e943e07bb36a2e4511b45996f8d4858a
Jul 16 00:51:23.000887 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:51:23.000852    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 16 00:51:23.881103 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:51:23.881058    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-z6sx5" event={"ID":"b2d0e466-a859-4e50-8703-43adcdcb1aaa","Type":"ContainerStarted","Data":"9df94d319888785c7b89b1c2ad472df8e943e07bb36a2e4511b45996f8d4858a"}
Jul 16 00:51:24.746683 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:51:24.746642    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl7fx"]
Jul 16 00:51:24.887523 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:51:24.887485    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-z6sx5" event={"ID":"b2d0e466-a859-4e50-8703-43adcdcb1aaa","Type":"ContainerStarted","Data":"ec805ae7559f49b65e15f85bda7eb569c6f27854867d314753c7a87063edc869"}
Jul 16 00:51:24.904240 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:51:24.904188    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-z6sx5" podStartSLOduration=1.901985565 podStartE2EDuration="2.904170003s" totalImagesPullingTime="1.002184438s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 00:51:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 00:51:24.903082678 +0000 UTC m=+657.338344415" watchObservedRunningTime="2026-07-16 00:51:24.904170003 +0000 UTC m=+657.339431720"
Jul 16 00:51:32.850922 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:51:32.850856    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-z6sx5"
Jul 16 00:51:32.851322 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:51:32.850940    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-z6sx5"
Jul 16 00:51:32.864262 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:51:32.864232    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-z6sx5"
Jul 16 00:51:32.936579 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:51:32.936553    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-z6sx5"
Jul 16 00:51:33.044815 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:51:33.044783    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-fzg7k"]
Jul 16 00:51:33.048229 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:51:33.048204    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-fzg7k"
Jul 16 00:51:33.050099 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:51:33.050078    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Jul 16 00:51:33.062212 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:51:33.062185    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-fzg7k"]
Jul 16 00:51:33.179586 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:51:33.179490    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/343c8ea6-28e1-44cc-aa55-41d91d40fbe1-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-fzg7k\" (UID: \"343c8ea6-28e1-44cc-aa55-41d91d40fbe1\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-fzg7k"
Jul 16 00:51:33.179586 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:51:33.179549    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/343c8ea6-28e1-44cc-aa55-41d91d40fbe1-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-fzg7k\" (UID: \"343c8ea6-28e1-44cc-aa55-41d91d40fbe1\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-fzg7k"
Jul 16 00:51:33.179801 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:51:33.179624    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/343c8ea6-28e1-44cc-aa55-41d91d40fbe1-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-fzg7k\" (UID: \"343c8ea6-28e1-44cc-aa55-41d91d40fbe1\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-fzg7k"
Jul 16 00:51:33.179801 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:51:33.179670    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/343c8ea6-28e1-44cc-aa55-41d91d40fbe1-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-fzg7k\" (UID: \"343c8ea6-28e1-44cc-aa55-41d91d40fbe1\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-fzg7k"
Jul 16 00:51:33.179801 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:51:33.179692    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-298df\" (UniqueName: \"kubernetes.io/projected/343c8ea6-28e1-44cc-aa55-41d91d40fbe1-kube-api-access-298df\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-fzg7k\" (UID: \"343c8ea6-28e1-44cc-aa55-41d91d40fbe1\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-fzg7k"
Jul 16 00:51:33.179801 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:51:33.179743    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/343c8ea6-28e1-44cc-aa55-41d91d40fbe1-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-fzg7k\" (UID: \"343c8ea6-28e1-44cc-aa55-41d91d40fbe1\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-fzg7k"
Jul 16 00:51:33.281162 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:51:33.281117    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/343c8ea6-28e1-44cc-aa55-41d91d40fbe1-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-fzg7k\" (UID: \"343c8ea6-28e1-44cc-aa55-41d91d40fbe1\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-fzg7k"
Jul 16 00:51:33.281348 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:51:33.281174    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/343c8ea6-28e1-44cc-aa55-41d91d40fbe1-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-fzg7k\" (UID: \"343c8ea6-28e1-44cc-aa55-41d91d40fbe1\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-fzg7k"
Jul 16 00:51:33.281348 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:51:33.281214    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/343c8ea6-28e1-44cc-aa55-41d91d40fbe1-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-fzg7k\" (UID: \"343c8ea6-28e1-44cc-aa55-41d91d40fbe1\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-fzg7k"
Jul 16 00:51:33.281348 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:51:33.281240    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-298df\" (UniqueName: \"kubernetes.io/projected/343c8ea6-28e1-44cc-aa55-41d91d40fbe1-kube-api-access-298df\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-fzg7k\" (UID: \"343c8ea6-28e1-44cc-aa55-41d91d40fbe1\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-fzg7k"
Jul 16 00:51:33.281348 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:51:33.281268    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/343c8ea6-28e1-44cc-aa55-41d91d40fbe1-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-fzg7k\" (UID: \"343c8ea6-28e1-44cc-aa55-41d91d40fbe1\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-fzg7k"
Jul 16 00:51:33.281545 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:51:33.281468    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/343c8ea6-28e1-44cc-aa55-41d91d40fbe1-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-fzg7k\" (UID: \"343c8ea6-28e1-44cc-aa55-41d91d40fbe1\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-fzg7k"
Jul 16 00:51:33.281715 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:51:33.281693    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/343c8ea6-28e1-44cc-aa55-41d91d40fbe1-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-fzg7k\" (UID: \"343c8ea6-28e1-44cc-aa55-41d91d40fbe1\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-fzg7k"
Jul 16 00:51:33.281822 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:51:33.281791    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/343c8ea6-28e1-44cc-aa55-41d91d40fbe1-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-fzg7k\" (UID: \"343c8ea6-28e1-44cc-aa55-41d91d40fbe1\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-fzg7k"
Jul 16 00:51:33.281910 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:51:33.281817    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/343c8ea6-28e1-44cc-aa55-41d91d40fbe1-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-fzg7k\" (UID: \"343c8ea6-28e1-44cc-aa55-41d91d40fbe1\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-fzg7k"
Jul 16 00:51:33.283417 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:51:33.283399    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/343c8ea6-28e1-44cc-aa55-41d91d40fbe1-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-fzg7k\" (UID: \"343c8ea6-28e1-44cc-aa55-41d91d40fbe1\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-fzg7k"
Jul 16 00:51:33.283695 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:51:33.283676    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/343c8ea6-28e1-44cc-aa55-41d91d40fbe1-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-fzg7k\" (UID: \"343c8ea6-28e1-44cc-aa55-41d91d40fbe1\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-fzg7k"
Jul 16 00:51:33.288473 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:51:33.288450    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-298df\" (UniqueName: \"kubernetes.io/projected/343c8ea6-28e1-44cc-aa55-41d91d40fbe1-kube-api-access-298df\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-fzg7k\" (UID: \"343c8ea6-28e1-44cc-aa55-41d91d40fbe1\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-fzg7k"
Jul 16 00:51:33.357843 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:51:33.357802    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-fzg7k"
Jul 16 00:51:33.492437 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:51:33.492412    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-fzg7k"]
Jul 16 00:51:33.494447 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:51:33.494404    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod343c8ea6_28e1_44cc_aa55_41d91d40fbe1.slice/crio-f00ec774cf6eb276089a8a23437f8ef89fce09989cd17236e1c46ec7659edcd0 WatchSource:0}: Error finding container f00ec774cf6eb276089a8a23437f8ef89fce09989cd17236e1c46ec7659edcd0: Status 404 returned error can't find the container with id f00ec774cf6eb276089a8a23437f8ef89fce09989cd17236e1c46ec7659edcd0
Jul 16 00:51:33.930948 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:51:33.930899    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-fzg7k" event={"ID":"343c8ea6-28e1-44cc-aa55-41d91d40fbe1","Type":"ContainerStarted","Data":"6f5f5298f7a8f7b336bfbf697bf9d3fa18ca9c06841cddec9a7b18ad0a5e5436"}
Jul 16 00:51:33.930948 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:51:33.930956    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-fzg7k" event={"ID":"343c8ea6-28e1-44cc-aa55-41d91d40fbe1","Type":"ContainerStarted","Data":"f00ec774cf6eb276089a8a23437f8ef89fce09989cd17236e1c46ec7659edcd0"}
Jul 16 00:51:33.951412 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:51:33.951308    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-fzg7k" podStartSLOduration=0.779264471 podStartE2EDuration="951.291919ms" totalImagesPullingTime="172.027448ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 00:51:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 00:51:33.949904948 +0000 UTC m=+666.385166680" watchObservedRunningTime="2026-07-16 00:51:33.951291919 +0000 UTC m=+666.386553658"
Jul 16 00:51:36.039981 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:51:36.039941    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl7fx"]
Jul 16 00:51:43.358878 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:51:43.358838    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-fzg7k"
Jul 16 00:51:43.359310 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:51:43.358905    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-fzg7k"
Jul 16 00:51:43.371692 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:51:43.371666    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-fzg7k"
Jul 16 00:51:43.984686 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:51:43.984655    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-fzg7k"
Jul 16 00:52:39.722236 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:52:39.722200    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-c6466dcb6-jvh97"]
Jul 16 00:52:39.725646 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:52:39.725628    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c6466dcb6-jvh97"
Jul 16 00:52:39.728112 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:52:39.728088    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-7cmff\""
Jul 16 00:52:39.728238 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:52:39.728094    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Jul 16 00:52:39.734367 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:52:39.734337    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c6466dcb6-jvh97"]
Jul 16 00:52:39.792244 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:52:39.792202    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75d6h\" (UniqueName: \"kubernetes.io/projected/d06ed8c7-ae3c-4c21-83bd-fd781e9f23c8-kube-api-access-75d6h\") pod \"authorino-c6466dcb6-jvh97\" (UID: \"d06ed8c7-ae3c-4c21-83bd-fd781e9f23c8\") " pod="kuadrant-system/authorino-c6466dcb6-jvh97"
Jul 16 00:52:39.792422 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:52:39.792345    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d06ed8c7-ae3c-4c21-83bd-fd781e9f23c8-tls-cert\") pod \"authorino-c6466dcb6-jvh97\" (UID: \"d06ed8c7-ae3c-4c21-83bd-fd781e9f23c8\") " pod="kuadrant-system/authorino-c6466dcb6-jvh97"
Jul 16 00:52:39.893558 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:52:39.893516    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75d6h\" (UniqueName: \"kubernetes.io/projected/d06ed8c7-ae3c-4c21-83bd-fd781e9f23c8-kube-api-access-75d6h\") pod \"authorino-c6466dcb6-jvh97\" (UID: \"d06ed8c7-ae3c-4c21-83bd-fd781e9f23c8\") " pod="kuadrant-system/authorino-c6466dcb6-jvh97"
Jul 16 00:52:39.893749 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:52:39.893622    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d06ed8c7-ae3c-4c21-83bd-fd781e9f23c8-tls-cert\") pod \"authorino-c6466dcb6-jvh97\" (UID: \"d06ed8c7-ae3c-4c21-83bd-fd781e9f23c8\") " pod="kuadrant-system/authorino-c6466dcb6-jvh97"
Jul 16 00:52:39.896088 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:52:39.896064    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d06ed8c7-ae3c-4c21-83bd-fd781e9f23c8-tls-cert\") pod \"authorino-c6466dcb6-jvh97\" (UID: \"d06ed8c7-ae3c-4c21-83bd-fd781e9f23c8\") " pod="kuadrant-system/authorino-c6466dcb6-jvh97"
Jul 16 00:52:39.900796 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:52:39.900765    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-75d6h\" (UniqueName: \"kubernetes.io/projected/d06ed8c7-ae3c-4c21-83bd-fd781e9f23c8-kube-api-access-75d6h\") pod \"authorino-c6466dcb6-jvh97\" (UID: \"d06ed8c7-ae3c-4c21-83bd-fd781e9f23c8\") " pod="kuadrant-system/authorino-c6466dcb6-jvh97"
Jul 16 00:52:40.035785 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:52:40.035689    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c6466dcb6-jvh97"
Jul 16 00:52:40.176047 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:52:40.176018    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c6466dcb6-jvh97"]
Jul 16 00:52:40.178307 ip-10-0-143-65 kubenswrapper[2563]: W0716 00:52:40.178281    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd06ed8c7_ae3c_4c21_83bd_fd781e9f23c8.slice/crio-16960b2545eb4d034c10e824cfd062149f6812c3c05083eb96090cccd04aa728 WatchSource:0}: Error finding container 16960b2545eb4d034c10e824cfd062149f6812c3c05083eb96090cccd04aa728: Status 404 returned error can't find the container with id 16960b2545eb4d034c10e824cfd062149f6812c3c05083eb96090cccd04aa728
Jul 16 00:52:40.221987 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:52:40.221944    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c6466dcb6-jvh97" event={"ID":"d06ed8c7-ae3c-4c21-83bd-fd781e9f23c8","Type":"ContainerStarted","Data":"16960b2545eb4d034c10e824cfd062149f6812c3c05083eb96090cccd04aa728"}
Jul 16 00:52:41.230085 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:52:41.230043    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c6466dcb6-jvh97" event={"ID":"d06ed8c7-ae3c-4c21-83bd-fd781e9f23c8","Type":"ContainerStarted","Data":"92012eecc2476be4ee07f0abf0c5ad3105717c827ac17e7e016d0baa2b3530b9"}
Jul 16 00:52:41.250150 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:52:41.250095    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-c6466dcb6-jvh97" podStartSLOduration=1.868674664 podStartE2EDuration="2.250078467s" totalImagesPullingTime="381.403803ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 00:52:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 00:52:41.248191912 +0000 UTC m=+733.683453650" watchObservedRunningTime="2026-07-16 00:52:41.250078467 +0000 UTC m=+733.685340205"
Jul 16 00:53:10.659186 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:53:10.659151    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl7fx"]
Jul 16 00:53:20.642619 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:53:20.642581    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl7fx"]
Jul 16 00:53:32.173324 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:53:32.173282    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl7fx"]
Jul 16 00:53:41.761877 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:53:41.761817    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl7fx"]
Jul 16 00:53:52.055711 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:53:52.055615    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl7fx"]
Jul 16 00:54:19.042545 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:54:19.042510    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl7fx"]
Jul 16 00:54:31.260840 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:54:31.260797    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl7fx"]
Jul 16 00:55:09.939934 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:55:09.939889    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl7fx"]
Jul 16 00:55:19.145507 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:55:19.145468    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl7fx"]
Jul 16 00:55:27.845066 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:55:27.844985    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl7fx"]
Jul 16 00:55:27.917951 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:55:27.917926    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bn5hr_e990a21d-614b-4d1d-be54-0df6e44487ed/ovn-acl-logging/0.log"
Jul 16 00:55:27.918457 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:55:27.918437    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bn5hr_e990a21d-614b-4d1d-be54-0df6e44487ed/ovn-acl-logging/0.log"
Jul 16 00:55:37.149104 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:55:37.149067    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl7fx"]
Jul 16 00:55:47.253664 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:55:47.253628    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl7fx"]
Jul 16 00:55:56.642737 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:55:56.642699    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl7fx"]
Jul 16 00:56:08.849496 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:56:08.849460    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl7fx"]
Jul 16 00:56:18.744928 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:56:18.744890    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl7fx"]
Jul 16 00:56:40.549876 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:56:40.549816    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl7fx"]
Jul 16 00:56:51.949427 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:56:51.949340    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl7fx"]
Jul 16 00:57:14.047996 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:57:14.047949    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl7fx"]
Jul 16 00:57:26.243167 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:57:26.243130    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl7fx"]
Jul 16 00:57:41.557763 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:57:41.557726    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl7fx"]
Jul 16 00:57:52.956321 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:57:52.956285    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl7fx"]
Jul 16 00:58:02.958701 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:58:02.958651    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl7fx"]
Jul 16 00:58:12.544201 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:58:12.544163    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl7fx"]
Jul 16 00:58:21.753441 ip-10-0-143-65 kubenswrapper[2563]: I0716 00:58:21.753364    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl7fx"]
Jul 16 01:00:11.387361 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:00:11.387319    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl7fx"]
Jul 16 01:00:20.640319 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:00:20.640280    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl7fx"]
Jul 16 01:00:27.945971 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:00:27.945945    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bn5hr_e990a21d-614b-4d1d-be54-0df6e44487ed/ovn-acl-logging/0.log"
Jul 16 01:00:27.947053 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:00:27.947028    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bn5hr_e990a21d-614b-4d1d-be54-0df6e44487ed/ovn-acl-logging/0.log"
Jul 16 01:00:33.348312 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:00:33.348277    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl7fx"]
Jul 16 01:00:46.760526 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:00:46.760486    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl7fx"]
Jul 16 01:00:56.539582 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:00:56.539548    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl7fx"]
Jul 16 01:01:07.259216 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:01:07.259180    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl7fx"]
Jul 16 01:01:15.851624 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:01:15.851590    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl7fx"]
Jul 16 01:01:27.246144 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:01:27.246065    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl7fx"]
Jul 16 01:01:38.147445 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:01:38.147401    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl7fx"]
Jul 16 01:01:46.542197 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:01:46.542164    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl7fx"]
Jul 16 01:01:57.842133 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:01:57.842095    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl7fx"]
Jul 16 01:02:19.252837 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:02:19.252786    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl7fx"]
Jul 16 01:02:28.147715 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:02:28.147681    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl7fx"]
Jul 16 01:02:44.850644 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:02:44.850608    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl7fx"]
Jul 16 01:02:53.053203 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:02:53.053106    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl7fx"]
Jul 16 01:03:04.243319 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:03:04.243281    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl7fx"]
Jul 16 01:03:15.745150 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:03:15.745116    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl7fx"]
Jul 16 01:03:25.634090 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:03:25.634055    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl7fx"]
Jul 16 01:03:33.253815 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:03:33.253778    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl7fx"]
Jul 16 01:03:44.345011 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:03:44.344966    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl7fx"]
Jul 16 01:03:53.952677 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:03:53.952639    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl7fx"]
Jul 16 01:04:07.160699 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:04:07.160666    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl7fx"]
Jul 16 01:04:16.552253 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:04:16.552214    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl7fx"]
Jul 16 01:04:26.149522 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:04:26.149441    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl7fx"]
Jul 16 01:04:35.439809 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:04:35.439774    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl7fx"]
Jul 16 01:04:48.651654 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:04:48.651621    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl7fx"]
Jul 16 01:04:57.848126 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:04:57.848085    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl7fx"]
Jul 16 01:05:11.057908 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:05:11.057857    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl7fx"]
Jul 16 01:05:22.542714 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:05:22.542680    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl7fx"]
Jul 16 01:05:27.973120 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:05:27.973094    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bn5hr_e990a21d-614b-4d1d-be54-0df6e44487ed/ovn-acl-logging/0.log"
Jul 16 01:05:27.981765 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:05:27.981741    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bn5hr_e990a21d-614b-4d1d-be54-0df6e44487ed/ovn-acl-logging/0.log"
Jul 16 01:05:37.647507 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:05:37.647474    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl7fx"]
Jul 16 01:05:51.038185 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:05:51.038112    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl7fx"]
Jul 16 01:06:00.346261 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:06:00.346226    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl7fx"]
Jul 16 01:06:09.959456 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:06:09.959422    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl7fx"]
Jul 16 01:06:29.068208 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:06:29.068174    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl7fx"]
Jul 16 01:06:41.061105 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:06:41.061070    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl7fx"]
Jul 16 01:06:54.200068 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:06:54.200032    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl7fx"]
Jul 16 01:08:17.303901 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:08:17.303852    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl7fx"]
Jul 16 01:08:26.554188 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:08:26.554154    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl7fx"]
Jul 16 01:08:37.647675 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:08:37.647642    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl7fx"]
Jul 16 01:08:55.858047 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:08:55.857967    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl7fx"]
Jul 16 01:09:18.549452 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:09:18.549415    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl7fx"]
Jul 16 01:09:29.673250 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:09:29.673213    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl7fx"]
Jul 16 01:10:12.047197 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:10:12.047144    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl7fx"]
Jul 16 01:10:28.011209 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:10:28.011179    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bn5hr_e990a21d-614b-4d1d-be54-0df6e44487ed/ovn-acl-logging/0.log"
Jul 16 01:10:28.014602 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:10:28.014583    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bn5hr_e990a21d-614b-4d1d-be54-0df6e44487ed/ovn-acl-logging/0.log"
Jul 16 01:10:32.546114 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:10:32.546074    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl7fx"]
Jul 16 01:10:36.065808 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:10:36.065732    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-7d5e85/test-model-7d5e85-kserve-69f9954f77-lkczd"]
Jul 16 01:10:36.069101 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:10:36.069081    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-7d5e85/test-model-7d5e85-kserve-69f9954f77-lkczd"
Jul 16 01:10:36.071667 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:10:36.071645    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-7d5e85\"/\"default-dockercfg-2j95h\""
Jul 16 01:10:36.071792 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:10:36.071670    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-7d5e85\"/\"kube-root-ca.crt\""
Jul 16 01:10:36.071792 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:10:36.071728    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-7d5e85\"/\"test-model-7d5e85-kserve-self-signed-certs\""
Jul 16 01:10:36.071792 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:10:36.071670    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-7d5e85\"/\"openshift-service-ca.crt\""
Jul 16 01:10:36.079000 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:10:36.078972    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-7d5e85/test-model-7d5e85-kserve-69f9954f77-lkczd"]
Jul 16 01:10:36.091588 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:10:36.091559    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e5421e0-6bdd-4434-8e78-d0caf0ceded9-tls-certs\") pod \"test-model-7d5e85-kserve-69f9954f77-lkczd\" (UID: \"6e5421e0-6bdd-4434-8e78-d0caf0ceded9\") " pod="ai-tenant-e2e-inf-a-7d5e85/test-model-7d5e85-kserve-69f9954f77-lkczd"
Jul 16 01:10:36.091765 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:10:36.091594    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6jnq\" (UniqueName: \"kubernetes.io/projected/6e5421e0-6bdd-4434-8e78-d0caf0ceded9-kube-api-access-m6jnq\") pod \"test-model-7d5e85-kserve-69f9954f77-lkczd\" (UID: \"6e5421e0-6bdd-4434-8e78-d0caf0ceded9\") " pod="ai-tenant-e2e-inf-a-7d5e85/test-model-7d5e85-kserve-69f9954f77-lkczd"
Jul 16 01:10:36.091765 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:10:36.091710    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e5421e0-6bdd-4434-8e78-d0caf0ceded9-home\") pod \"test-model-7d5e85-kserve-69f9954f77-lkczd\" (UID: \"6e5421e0-6bdd-4434-8e78-d0caf0ceded9\") " pod="ai-tenant-e2e-inf-a-7d5e85/test-model-7d5e85-kserve-69f9954f77-lkczd"
Jul 16 01:10:36.091921 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:10:36.091791    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e5421e0-6bdd-4434-8e78-d0caf0ceded9-model-cache\") pod \"test-model-7d5e85-kserve-69f9954f77-lkczd\" (UID: \"6e5421e0-6bdd-4434-8e78-d0caf0ceded9\") " pod="ai-tenant-e2e-inf-a-7d5e85/test-model-7d5e85-kserve-69f9954f77-lkczd"
Jul 16 01:10:36.091979 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:10:36.091921    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e5421e0-6bdd-4434-8e78-d0caf0ceded9-tmp-dir\") pod \"test-model-7d5e85-kserve-69f9954f77-lkczd\" (UID: \"6e5421e0-6bdd-4434-8e78-d0caf0ceded9\") " pod="ai-tenant-e2e-inf-a-7d5e85/test-model-7d5e85-kserve-69f9954f77-lkczd"
Jul 16 01:10:36.092028 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:10:36.091990    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e5421e0-6bdd-4434-8e78-d0caf0ceded9-dshm\") pod \"test-model-7d5e85-kserve-69f9954f77-lkczd\" (UID: \"6e5421e0-6bdd-4434-8e78-d0caf0ceded9\") " pod="ai-tenant-e2e-inf-a-7d5e85/test-model-7d5e85-kserve-69f9954f77-lkczd"
Jul 16 01:10:36.192765 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:10:36.192730    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e5421e0-6bdd-4434-8e78-d0caf0ceded9-home\") pod \"test-model-7d5e85-kserve-69f9954f77-lkczd\" (UID: \"6e5421e0-6bdd-4434-8e78-d0caf0ceded9\") " pod="ai-tenant-e2e-inf-a-7d5e85/test-model-7d5e85-kserve-69f9954f77-lkczd"
Jul 16 01:10:36.192988 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:10:36.192790    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e5421e0-6bdd-4434-8e78-d0caf0ceded9-model-cache\") pod \"test-model-7d5e85-kserve-69f9954f77-lkczd\" (UID: \"6e5421e0-6bdd-4434-8e78-d0caf0ceded9\") " pod="ai-tenant-e2e-inf-a-7d5e85/test-model-7d5e85-kserve-69f9954f77-lkczd"
Jul 16 01:10:36.192988 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:10:36.192824    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e5421e0-6bdd-4434-8e78-d0caf0ceded9-tmp-dir\") pod \"test-model-7d5e85-kserve-69f9954f77-lkczd\" (UID: \"6e5421e0-6bdd-4434-8e78-d0caf0ceded9\") " pod="ai-tenant-e2e-inf-a-7d5e85/test-model-7d5e85-kserve-69f9954f77-lkczd"
Jul 16 01:10:36.192988 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:10:36.192852    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e5421e0-6bdd-4434-8e78-d0caf0ceded9-dshm\") pod \"test-model-7d5e85-kserve-69f9954f77-lkczd\" (UID: \"6e5421e0-6bdd-4434-8e78-d0caf0ceded9\") " pod="ai-tenant-e2e-inf-a-7d5e85/test-model-7d5e85-kserve-69f9954f77-lkczd"
Jul 16 01:10:36.192988 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:10:36.192905    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e5421e0-6bdd-4434-8e78-d0caf0ceded9-tls-certs\") pod \"test-model-7d5e85-kserve-69f9954f77-lkczd\" (UID: \"6e5421e0-6bdd-4434-8e78-d0caf0ceded9\") " pod="ai-tenant-e2e-inf-a-7d5e85/test-model-7d5e85-kserve-69f9954f77-lkczd"
Jul 16 01:10:36.192988 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:10:36.192929    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6jnq\" (UniqueName: \"kubernetes.io/projected/6e5421e0-6bdd-4434-8e78-d0caf0ceded9-kube-api-access-m6jnq\") pod \"test-model-7d5e85-kserve-69f9954f77-lkczd\" (UID: \"6e5421e0-6bdd-4434-8e78-d0caf0ceded9\") " pod="ai-tenant-e2e-inf-a-7d5e85/test-model-7d5e85-kserve-69f9954f77-lkczd"
Jul 16 01:10:36.193277 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:10:36.193253    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e5421e0-6bdd-4434-8e78-d0caf0ceded9-home\") pod \"test-model-7d5e85-kserve-69f9954f77-lkczd\" (UID: \"6e5421e0-6bdd-4434-8e78-d0caf0ceded9\") " pod="ai-tenant-e2e-inf-a-7d5e85/test-model-7d5e85-kserve-69f9954f77-lkczd"
Jul 16 01:10:36.193350 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:10:36.193286    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e5421e0-6bdd-4434-8e78-d0caf0ceded9-model-cache\") pod \"test-model-7d5e85-kserve-69f9954f77-lkczd\" (UID: \"6e5421e0-6bdd-4434-8e78-d0caf0ceded9\") " pod="ai-tenant-e2e-inf-a-7d5e85/test-model-7d5e85-kserve-69f9954f77-lkczd"
Jul 16 01:10:36.193350 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:10:36.193319    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e5421e0-6bdd-4434-8e78-d0caf0ceded9-tmp-dir\") pod \"test-model-7d5e85-kserve-69f9954f77-lkczd\" (UID: \"6e5421e0-6bdd-4434-8e78-d0caf0ceded9\") " pod="ai-tenant-e2e-inf-a-7d5e85/test-model-7d5e85-kserve-69f9954f77-lkczd"
Jul 16 01:10:36.195268 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:10:36.195251    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e5421e0-6bdd-4434-8e78-d0caf0ceded9-dshm\") pod \"test-model-7d5e85-kserve-69f9954f77-lkczd\" (UID: \"6e5421e0-6bdd-4434-8e78-d0caf0ceded9\") " pod="ai-tenant-e2e-inf-a-7d5e85/test-model-7d5e85-kserve-69f9954f77-lkczd"
Jul 16 01:10:36.195510 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:10:36.195490    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e5421e0-6bdd-4434-8e78-d0caf0ceded9-tls-certs\") pod \"test-model-7d5e85-kserve-69f9954f77-lkczd\" (UID: \"6e5421e0-6bdd-4434-8e78-d0caf0ceded9\") " pod="ai-tenant-e2e-inf-a-7d5e85/test-model-7d5e85-kserve-69f9954f77-lkczd"
Jul 16 01:10:36.201279 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:10:36.201255    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6jnq\" (UniqueName: \"kubernetes.io/projected/6e5421e0-6bdd-4434-8e78-d0caf0ceded9-kube-api-access-m6jnq\") pod \"test-model-7d5e85-kserve-69f9954f77-lkczd\" (UID: \"6e5421e0-6bdd-4434-8e78-d0caf0ceded9\") " pod="ai-tenant-e2e-inf-a-7d5e85/test-model-7d5e85-kserve-69f9954f77-lkczd"
Jul 16 01:10:36.380512 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:10:36.380475    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-7d5e85/test-model-7d5e85-kserve-69f9954f77-lkczd"
Jul 16 01:10:36.517983 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:10:36.517946    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-7d5e85/test-model-7d5e85-kserve-69f9954f77-lkczd"]
Jul 16 01:10:36.520604 ip-10-0-143-65 kubenswrapper[2563]: W0716 01:10:36.520579    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6e5421e0_6bdd_4434_8e78_d0caf0ceded9.slice/crio-24e28f9f6477a6ce900092474ffeb48ab0eac7b178d00bf9c3461ea38706e29b WatchSource:0}: Error finding container 24e28f9f6477a6ce900092474ffeb48ab0eac7b178d00bf9c3461ea38706e29b: Status 404 returned error can't find the container with id 24e28f9f6477a6ce900092474ffeb48ab0eac7b178d00bf9c3461ea38706e29b
Jul 16 01:10:37.125365 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:10:37.125321    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-7d5e85/test-model-7d5e85-kserve-69f9954f77-lkczd" event={"ID":"6e5421e0-6bdd-4434-8e78-d0caf0ceded9","Type":"ContainerStarted","Data":"515412771dbeacf3c71975ffbdeabe3fd5cd1863af227f927f1afc2b6411b3a9"}
Jul 16 01:10:37.125777 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:10:37.125371    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-7d5e85/test-model-7d5e85-kserve-69f9954f77-lkczd" event={"ID":"6e5421e0-6bdd-4434-8e78-d0caf0ceded9","Type":"ContainerStarted","Data":"24e28f9f6477a6ce900092474ffeb48ab0eac7b178d00bf9c3461ea38706e29b"}
Jul 16 01:10:37.145072 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:10:37.144992    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-7d5e85/test-model-7d5e85-kserve-69f9954f77-lkczd" podStartSLOduration=1.144974271 podStartE2EDuration="1.144974271s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 01:10:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 01:10:37.143995521 +0000 UTC m=+1809.579257260" watchObservedRunningTime="2026-07-16 01:10:37.144974271 +0000 UTC m=+1809.580236011"
Jul 16 01:10:43.146111 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:10:43.146077    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl7fx"]
Jul 16 01:10:46.381459 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:10:46.381424    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-7d5e85/test-model-7d5e85-kserve-69f9954f77-lkczd"
Jul 16 01:10:46.381459 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:10:46.381465    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-7d5e85/test-model-7d5e85-kserve-69f9954f77-lkczd"
Jul 16 01:10:46.393849 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:10:46.393824    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-7d5e85/test-model-7d5e85-kserve-69f9954f77-lkczd"
Jul 16 01:10:47.184916 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:10:47.184880    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-7d5e85/test-model-7d5e85-kserve-69f9954f77-lkczd"
Jul 16 01:11:17.351981 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:11:17.351944    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-e5854b/test-model-e5854b-kserve-5d5d89768c-xv8rd"]
Jul 16 01:11:17.355710 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:11:17.355689    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-e5854b/test-model-e5854b-kserve-5d5d89768c-xv8rd"
Jul 16 01:11:17.360016 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:11:17.359993    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-e5854b\"/\"kube-root-ca.crt\""
Jul 16 01:11:17.360154 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:11:17.360037    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-e5854b\"/\"default-dockercfg-rbgx8\""
Jul 16 01:11:17.360154 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:11:17.360057    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-e5854b\"/\"openshift-service-ca.crt\""
Jul 16 01:11:17.360154 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:11:17.360057    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-e5854b\"/\"test-model-e5854b-kserve-self-signed-certs\""
Jul 16 01:11:17.364841 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:11:17.364818    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-e5854b/test-model-e5854b-kserve-5d5d89768c-xv8rd"]
Jul 16 01:11:17.369627 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:11:17.369606    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9631b12e-5403-4e99-ac8c-f82116234653-tmp-dir\") pod \"test-model-e5854b-kserve-5d5d89768c-xv8rd\" (UID: \"9631b12e-5403-4e99-ac8c-f82116234653\") " pod="ai-tenant-e2e-inf-b-e5854b/test-model-e5854b-kserve-5d5d89768c-xv8rd"
Jul 16 01:11:17.369712 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:11:17.369638    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9631b12e-5403-4e99-ac8c-f82116234653-home\") pod \"test-model-e5854b-kserve-5d5d89768c-xv8rd\" (UID: \"9631b12e-5403-4e99-ac8c-f82116234653\") " pod="ai-tenant-e2e-inf-b-e5854b/test-model-e5854b-kserve-5d5d89768c-xv8rd"
Jul 16 01:11:17.369712 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:11:17.369655    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9631b12e-5403-4e99-ac8c-f82116234653-tls-certs\") pod \"test-model-e5854b-kserve-5d5d89768c-xv8rd\" (UID: \"9631b12e-5403-4e99-ac8c-f82116234653\") " pod="ai-tenant-e2e-inf-b-e5854b/test-model-e5854b-kserve-5d5d89768c-xv8rd"
Jul 16 01:11:17.369712 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:11:17.369684    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmwhq\" (UniqueName: \"kubernetes.io/projected/9631b12e-5403-4e99-ac8c-f82116234653-kube-api-access-wmwhq\") pod \"test-model-e5854b-kserve-5d5d89768c-xv8rd\" (UID: \"9631b12e-5403-4e99-ac8c-f82116234653\") " pod="ai-tenant-e2e-inf-b-e5854b/test-model-e5854b-kserve-5d5d89768c-xv8rd"
Jul 16 01:11:17.369813 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:11:17.369737    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9631b12e-5403-4e99-ac8c-f82116234653-model-cache\") pod \"test-model-e5854b-kserve-5d5d89768c-xv8rd\" (UID: \"9631b12e-5403-4e99-ac8c-f82116234653\") " pod="ai-tenant-e2e-inf-b-e5854b/test-model-e5854b-kserve-5d5d89768c-xv8rd"
Jul 16 01:11:17.369846 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:11:17.369824    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9631b12e-5403-4e99-ac8c-f82116234653-dshm\") pod \"test-model-e5854b-kserve-5d5d89768c-xv8rd\" (UID: \"9631b12e-5403-4e99-ac8c-f82116234653\") " pod="ai-tenant-e2e-inf-b-e5854b/test-model-e5854b-kserve-5d5d89768c-xv8rd"
Jul 16 01:11:17.471180 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:11:17.471145    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9631b12e-5403-4e99-ac8c-f82116234653-dshm\") pod \"test-model-e5854b-kserve-5d5d89768c-xv8rd\" (UID: \"9631b12e-5403-4e99-ac8c-f82116234653\") " pod="ai-tenant-e2e-inf-b-e5854b/test-model-e5854b-kserve-5d5d89768c-xv8rd"
Jul 16 01:11:17.471362 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:11:17.471213    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9631b12e-5403-4e99-ac8c-f82116234653-tmp-dir\") pod \"test-model-e5854b-kserve-5d5d89768c-xv8rd\" (UID: \"9631b12e-5403-4e99-ac8c-f82116234653\") " pod="ai-tenant-e2e-inf-b-e5854b/test-model-e5854b-kserve-5d5d89768c-xv8rd"
Jul 16 01:11:17.471362 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:11:17.471239    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9631b12e-5403-4e99-ac8c-f82116234653-home\") pod \"test-model-e5854b-kserve-5d5d89768c-xv8rd\" (UID: \"9631b12e-5403-4e99-ac8c-f82116234653\") " pod="ai-tenant-e2e-inf-b-e5854b/test-model-e5854b-kserve-5d5d89768c-xv8rd"
Jul 16 01:11:17.471362 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:11:17.471255    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9631b12e-5403-4e99-ac8c-f82116234653-tls-certs\") pod \"test-model-e5854b-kserve-5d5d89768c-xv8rd\" (UID: \"9631b12e-5403-4e99-ac8c-f82116234653\") " pod="ai-tenant-e2e-inf-b-e5854b/test-model-e5854b-kserve-5d5d89768c-xv8rd"
Jul 16 01:11:17.471362 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:11:17.471279    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmwhq\" (UniqueName: \"kubernetes.io/projected/9631b12e-5403-4e99-ac8c-f82116234653-kube-api-access-wmwhq\") pod \"test-model-e5854b-kserve-5d5d89768c-xv8rd\" (UID: \"9631b12e-5403-4e99-ac8c-f82116234653\") " pod="ai-tenant-e2e-inf-b-e5854b/test-model-e5854b-kserve-5d5d89768c-xv8rd"
Jul 16 01:11:17.471362 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:11:17.471308    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9631b12e-5403-4e99-ac8c-f82116234653-model-cache\") pod \"test-model-e5854b-kserve-5d5d89768c-xv8rd\" (UID: \"9631b12e-5403-4e99-ac8c-f82116234653\") " pod="ai-tenant-e2e-inf-b-e5854b/test-model-e5854b-kserve-5d5d89768c-xv8rd"
Jul 16 01:11:17.471711 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:11:17.471678    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9631b12e-5403-4e99-ac8c-f82116234653-home\") pod \"test-model-e5854b-kserve-5d5d89768c-xv8rd\" (UID: \"9631b12e-5403-4e99-ac8c-f82116234653\") " pod="ai-tenant-e2e-inf-b-e5854b/test-model-e5854b-kserve-5d5d89768c-xv8rd"
Jul 16 01:11:17.471711 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:11:17.471702    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9631b12e-5403-4e99-ac8c-f82116234653-tmp-dir\") pod \"test-model-e5854b-kserve-5d5d89768c-xv8rd\" (UID: \"9631b12e-5403-4e99-ac8c-f82116234653\") " pod="ai-tenant-e2e-inf-b-e5854b/test-model-e5854b-kserve-5d5d89768c-xv8rd"
Jul 16 01:11:17.471896 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:11:17.471775    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9631b12e-5403-4e99-ac8c-f82116234653-model-cache\") pod \"test-model-e5854b-kserve-5d5d89768c-xv8rd\" (UID: \"9631b12e-5403-4e99-ac8c-f82116234653\") " pod="ai-tenant-e2e-inf-b-e5854b/test-model-e5854b-kserve-5d5d89768c-xv8rd"
Jul 16 01:11:17.473700 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:11:17.473680    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9631b12e-5403-4e99-ac8c-f82116234653-tls-certs\") pod \"test-model-e5854b-kserve-5d5d89768c-xv8rd\" (UID: \"9631b12e-5403-4e99-ac8c-f82116234653\") " pod="ai-tenant-e2e-inf-b-e5854b/test-model-e5854b-kserve-5d5d89768c-xv8rd"
Jul 16 01:11:17.473801 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:11:17.473782    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9631b12e-5403-4e99-ac8c-f82116234653-dshm\") pod \"test-model-e5854b-kserve-5d5d89768c-xv8rd\" (UID: \"9631b12e-5403-4e99-ac8c-f82116234653\") " pod="ai-tenant-e2e-inf-b-e5854b/test-model-e5854b-kserve-5d5d89768c-xv8rd"
Jul 16 01:11:17.479435 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:11:17.479411    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmwhq\" (UniqueName: \"kubernetes.io/projected/9631b12e-5403-4e99-ac8c-f82116234653-kube-api-access-wmwhq\") pod \"test-model-e5854b-kserve-5d5d89768c-xv8rd\" (UID: \"9631b12e-5403-4e99-ac8c-f82116234653\") " pod="ai-tenant-e2e-inf-b-e5854b/test-model-e5854b-kserve-5d5d89768c-xv8rd"
Jul 16 01:11:17.665969 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:11:17.665879    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-e5854b/test-model-e5854b-kserve-5d5d89768c-xv8rd"
Jul 16 01:11:17.800106 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:11:17.799917    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-e5854b/test-model-e5854b-kserve-5d5d89768c-xv8rd"]
Jul 16 01:11:17.803015 ip-10-0-143-65 kubenswrapper[2563]: W0716 01:11:17.802986    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9631b12e_5403_4e99_ac8c_f82116234653.slice/crio-31d96299aef169276e5b9dfd8a79a18bc59521bcf91d6193b9123461de702c84 WatchSource:0}: Error finding container 31d96299aef169276e5b9dfd8a79a18bc59521bcf91d6193b9123461de702c84: Status 404 returned error can't find the container with id 31d96299aef169276e5b9dfd8a79a18bc59521bcf91d6193b9123461de702c84
Jul 16 01:11:18.314082 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:11:18.314043    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-e5854b/test-model-e5854b-kserve-5d5d89768c-xv8rd" event={"ID":"9631b12e-5403-4e99-ac8c-f82116234653","Type":"ContainerStarted","Data":"cee7a2d567b2121e0df0b02d6437d9bc5329bbcbc59d4b7b689587d9e9339681"}
Jul 16 01:11:18.314256 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:11:18.314090    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-e5854b/test-model-e5854b-kserve-5d5d89768c-xv8rd" event={"ID":"9631b12e-5403-4e99-ac8c-f82116234653","Type":"ContainerStarted","Data":"31d96299aef169276e5b9dfd8a79a18bc59521bcf91d6193b9123461de702c84"}
Jul 16 01:11:18.334379 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:11:18.334321    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-e5854b/test-model-e5854b-kserve-5d5d89768c-xv8rd" podStartSLOduration=1.334303037 podStartE2EDuration="1.334303037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 01:11:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 01:11:18.333178775 +0000 UTC m=+1850.768440513" watchObservedRunningTime="2026-07-16 01:11:18.334303037 +0000 UTC m=+1850.769564775"
Jul 16 01:11:21.649975 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:11:21.649935    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl7fx"]
Jul 16 01:11:27.666997 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:11:27.666957    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-e5854b/test-model-e5854b-kserve-5d5d89768c-xv8rd"
Jul 16 01:11:27.667526 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:11:27.667240    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-e5854b/test-model-e5854b-kserve-5d5d89768c-xv8rd"
Jul 16 01:11:27.679559 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:11:27.679531    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-e5854b/test-model-e5854b-kserve-5d5d89768c-xv8rd"
Jul 16 01:11:28.369879 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:11:28.369835    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-e5854b/test-model-e5854b-kserve-5d5d89768c-xv8rd"
Jul 16 01:11:37.147851 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:11:37.147810    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl7fx"]
Jul 16 01:11:42.405288 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:11:42.405254    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-7d5e85/test-model-7d5e85-kserve-69f9954f77-lkczd"]
Jul 16 01:11:43.380609 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:11:43.380581    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-e5854b/test-model-e5854b-kserve-5d5d89768c-xv8rd"]
Jul 16 01:11:50.846599 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:11:50.846513    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl7fx"]
Jul 16 01:11:57.425055 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:11:57.424997    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-7d5e85/test-model-7d5e85-kserve-69f9954f77-lkczd" podUID="6e5421e0-6bdd-4434-8e78-d0caf0ceded9" containerName="main" containerID="cri-o://515412771dbeacf3c71975ffbdeabe3fd5cd1863af227f927f1afc2b6411b3a9" gracePeriod=45
Jul 16 01:11:57.679850 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:11:57.679775    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-7d5e85/test-model-7d5e85-kserve-69f9954f77-lkczd"
Jul 16 01:11:57.748158 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:11:57.748126    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e5421e0-6bdd-4434-8e78-d0caf0ceded9-model-cache\") pod \"6e5421e0-6bdd-4434-8e78-d0caf0ceded9\" (UID: \"6e5421e0-6bdd-4434-8e78-d0caf0ceded9\") "
Jul 16 01:11:57.748324 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:11:57.748181    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e5421e0-6bdd-4434-8e78-d0caf0ceded9-tmp-dir\") pod \"6e5421e0-6bdd-4434-8e78-d0caf0ceded9\" (UID: \"6e5421e0-6bdd-4434-8e78-d0caf0ceded9\") "
Jul 16 01:11:57.748324 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:11:57.748206    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e5421e0-6bdd-4434-8e78-d0caf0ceded9-dshm\") pod \"6e5421e0-6bdd-4434-8e78-d0caf0ceded9\" (UID: \"6e5421e0-6bdd-4434-8e78-d0caf0ceded9\") "
Jul 16 01:11:57.748324 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:11:57.748249    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e5421e0-6bdd-4434-8e78-d0caf0ceded9-tls-certs\") pod \"6e5421e0-6bdd-4434-8e78-d0caf0ceded9\" (UID: \"6e5421e0-6bdd-4434-8e78-d0caf0ceded9\") "
Jul 16 01:11:57.748324 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:11:57.748277    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m6jnq\" (UniqueName: \"kubernetes.io/projected/6e5421e0-6bdd-4434-8e78-d0caf0ceded9-kube-api-access-m6jnq\") pod \"6e5421e0-6bdd-4434-8e78-d0caf0ceded9\" (UID: \"6e5421e0-6bdd-4434-8e78-d0caf0ceded9\") "
Jul 16 01:11:57.748324 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:11:57.748298    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e5421e0-6bdd-4434-8e78-d0caf0ceded9-home\") pod \"6e5421e0-6bdd-4434-8e78-d0caf0ceded9\" (UID: \"6e5421e0-6bdd-4434-8e78-d0caf0ceded9\") "
Jul 16 01:11:57.748586 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:11:57.748400    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e5421e0-6bdd-4434-8e78-d0caf0ceded9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6e5421e0-6bdd-4434-8e78-d0caf0ceded9" (UID: "6e5421e0-6bdd-4434-8e78-d0caf0ceded9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 01:11:57.748586 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:11:57.748424    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e5421e0-6bdd-4434-8e78-d0caf0ceded9-model-cache" (OuterVolumeSpecName: "model-cache") pod "6e5421e0-6bdd-4434-8e78-d0caf0ceded9" (UID: "6e5421e0-6bdd-4434-8e78-d0caf0ceded9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 01:11:57.748699 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:11:57.748594    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e5421e0-6bdd-4434-8e78-d0caf0ceded9-model-cache\") on node \"ip-10-0-143-65.ec2.internal\" DevicePath \"\""
Jul 16 01:11:57.748699 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:11:57.748607    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e5421e0-6bdd-4434-8e78-d0caf0ceded9-tmp-dir\") on node \"ip-10-0-143-65.ec2.internal\" DevicePath \"\""
Jul 16 01:11:57.748699 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:11:57.748614    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e5421e0-6bdd-4434-8e78-d0caf0ceded9-home" (OuterVolumeSpecName: "home") pod "6e5421e0-6bdd-4434-8e78-d0caf0ceded9" (UID: "6e5421e0-6bdd-4434-8e78-d0caf0ceded9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 01:11:57.750487 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:11:57.750457    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e5421e0-6bdd-4434-8e78-d0caf0ceded9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6e5421e0-6bdd-4434-8e78-d0caf0ceded9" (UID: "6e5421e0-6bdd-4434-8e78-d0caf0ceded9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 01:11:57.750487 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:11:57.750458    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e5421e0-6bdd-4434-8e78-d0caf0ceded9-kube-api-access-m6jnq" (OuterVolumeSpecName: "kube-api-access-m6jnq") pod "6e5421e0-6bdd-4434-8e78-d0caf0ceded9" (UID: "6e5421e0-6bdd-4434-8e78-d0caf0ceded9"). InnerVolumeSpecName "kube-api-access-m6jnq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 01:11:57.750629 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:11:57.750535    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e5421e0-6bdd-4434-8e78-d0caf0ceded9-dshm" (OuterVolumeSpecName: "dshm") pod "6e5421e0-6bdd-4434-8e78-d0caf0ceded9" (UID: "6e5421e0-6bdd-4434-8e78-d0caf0ceded9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 01:11:57.849254 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:11:57.849220    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e5421e0-6bdd-4434-8e78-d0caf0ceded9-dshm\") on node \"ip-10-0-143-65.ec2.internal\" DevicePath \"\""
Jul 16 01:11:57.849254 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:11:57.849249    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e5421e0-6bdd-4434-8e78-d0caf0ceded9-tls-certs\") on node \"ip-10-0-143-65.ec2.internal\" DevicePath \"\""
Jul 16 01:11:57.849254 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:11:57.849261    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6jnq\" (UniqueName: \"kubernetes.io/projected/6e5421e0-6bdd-4434-8e78-d0caf0ceded9-kube-api-access-m6jnq\") on node \"ip-10-0-143-65.ec2.internal\" DevicePath \"\""
Jul 16 01:11:57.849474 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:11:57.849269    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e5421e0-6bdd-4434-8e78-d0caf0ceded9-home\") on node \"ip-10-0-143-65.ec2.internal\" DevicePath \"\""
Jul 16 01:11:58.400756 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:11:58.400706    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-e5854b/test-model-e5854b-kserve-5d5d89768c-xv8rd" podUID="9631b12e-5403-4e99-ac8c-f82116234653" containerName="main" containerID="cri-o://cee7a2d567b2121e0df0b02d6437d9bc5329bbcbc59d4b7b689587d9e9339681" gracePeriod=45
Jul 16 01:11:58.504386 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:11:58.504349    2563 generic.go:356] "Generic (PLEG): container finished" podID="6e5421e0-6bdd-4434-8e78-d0caf0ceded9" containerID="515412771dbeacf3c71975ffbdeabe3fd5cd1863af227f927f1afc2b6411b3a9" exitCode=0
Jul 16 01:11:58.504824 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:11:58.504405    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-7d5e85/test-model-7d5e85-kserve-69f9954f77-lkczd"
Jul 16 01:11:58.504824 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:11:58.504431    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-7d5e85/test-model-7d5e85-kserve-69f9954f77-lkczd" event={"ID":"6e5421e0-6bdd-4434-8e78-d0caf0ceded9","Type":"ContainerDied","Data":"515412771dbeacf3c71975ffbdeabe3fd5cd1863af227f927f1afc2b6411b3a9"}
Jul 16 01:11:58.504824 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:11:58.504475    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-7d5e85/test-model-7d5e85-kserve-69f9954f77-lkczd" event={"ID":"6e5421e0-6bdd-4434-8e78-d0caf0ceded9","Type":"ContainerDied","Data":"24e28f9f6477a6ce900092474ffeb48ab0eac7b178d00bf9c3461ea38706e29b"}
Jul 16 01:11:58.504824 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:11:58.504494    2563 scope.go:117] "RemoveContainer" containerID="515412771dbeacf3c71975ffbdeabe3fd5cd1863af227f927f1afc2b6411b3a9"
Jul 16 01:11:58.517553 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:11:58.517532    2563 scope.go:117] "RemoveContainer" containerID="515412771dbeacf3c71975ffbdeabe3fd5cd1863af227f927f1afc2b6411b3a9"
Jul 16 01:11:58.517821 ip-10-0-143-65 kubenswrapper[2563]: E0716 01:11:58.517799    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"515412771dbeacf3c71975ffbdeabe3fd5cd1863af227f927f1afc2b6411b3a9\": container with ID starting with 515412771dbeacf3c71975ffbdeabe3fd5cd1863af227f927f1afc2b6411b3a9 not found: ID does not exist" containerID="515412771dbeacf3c71975ffbdeabe3fd5cd1863af227f927f1afc2b6411b3a9"
Jul 16 01:11:58.517907 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:11:58.517829    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"515412771dbeacf3c71975ffbdeabe3fd5cd1863af227f927f1afc2b6411b3a9"} err="failed to get container status \"515412771dbeacf3c71975ffbdeabe3fd5cd1863af227f927f1afc2b6411b3a9\": rpc error: code = NotFound desc = could not find container \"515412771dbeacf3c71975ffbdeabe3fd5cd1863af227f927f1afc2b6411b3a9\": container with ID starting with 515412771dbeacf3c71975ffbdeabe3fd5cd1863af227f927f1afc2b6411b3a9 not found: ID does not exist"
Jul 16 01:11:58.523327 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:11:58.523304    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-7d5e85/test-model-7d5e85-kserve-69f9954f77-lkczd"]
Jul 16 01:11:58.532216 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:11:58.532196    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-7d5e85/test-model-7d5e85-kserve-69f9954f77-lkczd"]
Jul 16 01:11:58.654902 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:11:58.654821    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-e5854b/test-model-e5854b-kserve-5d5d89768c-xv8rd"
Jul 16 01:11:58.758510 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:11:58.758479    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9631b12e-5403-4e99-ac8c-f82116234653-home\") pod \"9631b12e-5403-4e99-ac8c-f82116234653\" (UID: \"9631b12e-5403-4e99-ac8c-f82116234653\") "
Jul 16 01:11:58.758756 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:11:58.758525    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wmwhq\" (UniqueName: \"kubernetes.io/projected/9631b12e-5403-4e99-ac8c-f82116234653-kube-api-access-wmwhq\") pod \"9631b12e-5403-4e99-ac8c-f82116234653\" (UID: \"9631b12e-5403-4e99-ac8c-f82116234653\") "
Jul 16 01:11:58.758756 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:11:58.758543    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9631b12e-5403-4e99-ac8c-f82116234653-model-cache\") pod \"9631b12e-5403-4e99-ac8c-f82116234653\" (UID: \"9631b12e-5403-4e99-ac8c-f82116234653\") "
Jul 16 01:11:58.758756 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:11:58.758606    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9631b12e-5403-4e99-ac8c-f82116234653-tls-certs\") pod \"9631b12e-5403-4e99-ac8c-f82116234653\" (UID: \"9631b12e-5403-4e99-ac8c-f82116234653\") "
Jul 16 01:11:58.758756 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:11:58.758637    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9631b12e-5403-4e99-ac8c-f82116234653-tmp-dir\") pod \"9631b12e-5403-4e99-ac8c-f82116234653\" (UID: \"9631b12e-5403-4e99-ac8c-f82116234653\") "
Jul 16 01:11:58.758756 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:11:58.758663    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9631b12e-5403-4e99-ac8c-f82116234653-dshm\") pod \"9631b12e-5403-4e99-ac8c-f82116234653\" (UID: \"9631b12e-5403-4e99-ac8c-f82116234653\") "
Jul 16 01:11:58.759052 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:11:58.758828    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9631b12e-5403-4e99-ac8c-f82116234653-home" (OuterVolumeSpecName: "home") pod "9631b12e-5403-4e99-ac8c-f82116234653" (UID: "9631b12e-5403-4e99-ac8c-f82116234653"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 01:11:58.759052 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:11:58.758889    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9631b12e-5403-4e99-ac8c-f82116234653-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9631b12e-5403-4e99-ac8c-f82116234653" (UID: "9631b12e-5403-4e99-ac8c-f82116234653"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 01:11:58.759052 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:11:58.758906    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9631b12e-5403-4e99-ac8c-f82116234653-model-cache" (OuterVolumeSpecName: "model-cache") pod "9631b12e-5403-4e99-ac8c-f82116234653" (UID: "9631b12e-5403-4e99-ac8c-f82116234653"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 01:11:58.759052 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:11:58.759013    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9631b12e-5403-4e99-ac8c-f82116234653-home\") on node \"ip-10-0-143-65.ec2.internal\" DevicePath \"\""
Jul 16 01:11:58.759052 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:11:58.759027    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9631b12e-5403-4e99-ac8c-f82116234653-model-cache\") on node \"ip-10-0-143-65.ec2.internal\" DevicePath \"\""
Jul 16 01:11:58.759052 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:11:58.759036    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9631b12e-5403-4e99-ac8c-f82116234653-tmp-dir\") on node \"ip-10-0-143-65.ec2.internal\" DevicePath \"\""
Jul 16 01:11:58.760822 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:11:58.760797    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9631b12e-5403-4e99-ac8c-f82116234653-dshm" (OuterVolumeSpecName: "dshm") pod "9631b12e-5403-4e99-ac8c-f82116234653" (UID: "9631b12e-5403-4e99-ac8c-f82116234653"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 01:11:58.760926 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:11:58.760838    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9631b12e-5403-4e99-ac8c-f82116234653-kube-api-access-wmwhq" (OuterVolumeSpecName: "kube-api-access-wmwhq") pod "9631b12e-5403-4e99-ac8c-f82116234653" (UID: "9631b12e-5403-4e99-ac8c-f82116234653"). InnerVolumeSpecName "kube-api-access-wmwhq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 01:11:58.760926 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:11:58.760855    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9631b12e-5403-4e99-ac8c-f82116234653-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9631b12e-5403-4e99-ac8c-f82116234653" (UID: "9631b12e-5403-4e99-ac8c-f82116234653"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 01:11:58.859778 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:11:58.859741    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmwhq\" (UniqueName: \"kubernetes.io/projected/9631b12e-5403-4e99-ac8c-f82116234653-kube-api-access-wmwhq\") on node \"ip-10-0-143-65.ec2.internal\" DevicePath \"\""
Jul 16 01:11:58.859778 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:11:58.859772    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9631b12e-5403-4e99-ac8c-f82116234653-tls-certs\") on node \"ip-10-0-143-65.ec2.internal\" DevicePath \"\""
Jul 16 01:11:58.859778 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:11:58.859782    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9631b12e-5403-4e99-ac8c-f82116234653-dshm\") on node \"ip-10-0-143-65.ec2.internal\" DevicePath \"\""
Jul 16 01:11:59.517216 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:11:59.517181    2563 generic.go:356] "Generic (PLEG): container finished" podID="9631b12e-5403-4e99-ac8c-f82116234653" containerID="cee7a2d567b2121e0df0b02d6437d9bc5329bbcbc59d4b7b689587d9e9339681" exitCode=0
Jul 16 01:11:59.517699 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:11:59.517231    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-e5854b/test-model-e5854b-kserve-5d5d89768c-xv8rd"
Jul 16 01:11:59.517699 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:11:59.517246    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-e5854b/test-model-e5854b-kserve-5d5d89768c-xv8rd" event={"ID":"9631b12e-5403-4e99-ac8c-f82116234653","Type":"ContainerDied","Data":"cee7a2d567b2121e0df0b02d6437d9bc5329bbcbc59d4b7b689587d9e9339681"}
Jul 16 01:11:59.517699 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:11:59.517283    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-e5854b/test-model-e5854b-kserve-5d5d89768c-xv8rd" event={"ID":"9631b12e-5403-4e99-ac8c-f82116234653","Type":"ContainerDied","Data":"31d96299aef169276e5b9dfd8a79a18bc59521bcf91d6193b9123461de702c84"}
Jul 16 01:11:59.517699 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:11:59.517306    2563 scope.go:117] "RemoveContainer" containerID="cee7a2d567b2121e0df0b02d6437d9bc5329bbcbc59d4b7b689587d9e9339681"
Jul 16 01:11:59.526033 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:11:59.525980    2563 scope.go:117] "RemoveContainer" containerID="cee7a2d567b2121e0df0b02d6437d9bc5329bbcbc59d4b7b689587d9e9339681"
Jul 16 01:11:59.526294 ip-10-0-143-65 kubenswrapper[2563]: E0716 01:11:59.526277    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cee7a2d567b2121e0df0b02d6437d9bc5329bbcbc59d4b7b689587d9e9339681\": container with ID starting with cee7a2d567b2121e0df0b02d6437d9bc5329bbcbc59d4b7b689587d9e9339681 not found: ID does not exist" containerID="cee7a2d567b2121e0df0b02d6437d9bc5329bbcbc59d4b7b689587d9e9339681"
Jul 16 01:11:59.526367 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:11:59.526301    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cee7a2d567b2121e0df0b02d6437d9bc5329bbcbc59d4b7b689587d9e9339681"} err="failed to get container status \"cee7a2d567b2121e0df0b02d6437d9bc5329bbcbc59d4b7b689587d9e9339681\": rpc error: code = NotFound desc = could not find container \"cee7a2d567b2121e0df0b02d6437d9bc5329bbcbc59d4b7b689587d9e9339681\": container with ID starting with cee7a2d567b2121e0df0b02d6437d9bc5329bbcbc59d4b7b689587d9e9339681 not found: ID does not exist"
Jul 16 01:11:59.538549 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:11:59.538525    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-e5854b/test-model-e5854b-kserve-5d5d89768c-xv8rd"]
Jul 16 01:11:59.541957 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:11:59.541934    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-e5854b/test-model-e5854b-kserve-5d5d89768c-xv8rd"]
Jul 16 01:12:00.005842 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:12:00.005806    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e5421e0-6bdd-4434-8e78-d0caf0ceded9" path="/var/lib/kubelet/pods/6e5421e0-6bdd-4434-8e78-d0caf0ceded9/volumes"
Jul 16 01:12:00.006263 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:12:00.006248    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9631b12e-5403-4e99-ac8c-f82116234653" path="/var/lib/kubelet/pods/9631b12e-5403-4e99-ac8c-f82116234653/volumes"
Jul 16 01:12:29.448137 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:12:29.448098    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl7fx"]
Jul 16 01:12:49.457344 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:12:49.457308    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl7fx"]
Jul 16 01:13:13.951971 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:13:13.951934    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl7fx"]
Jul 16 01:13:29.459815 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:13:29.459722    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl7fx"]
Jul 16 01:13:47.866571 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:13:47.866530    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl7fx"]
Jul 16 01:13:51.444516 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:13:51.444482    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-f9f50d/auth-test-model-f9f50d-kserve-769684b9d7-bd4lz"]
Jul 16 01:13:51.444897 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:13:51.444829    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e5421e0-6bdd-4434-8e78-d0caf0ceded9" containerName="main"
Jul 16 01:13:51.444897 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:13:51.444840    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e5421e0-6bdd-4434-8e78-d0caf0ceded9" containerName="main"
Jul 16 01:13:51.444897 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:13:51.444853    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9631b12e-5403-4e99-ac8c-f82116234653" containerName="main"
Jul 16 01:13:51.444897 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:13:51.444859    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="9631b12e-5403-4e99-ac8c-f82116234653" containerName="main"
Jul 16 01:13:51.445065 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:13:51.444937    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e5421e0-6bdd-4434-8e78-d0caf0ceded9" containerName="main"
Jul 16 01:13:51.445065 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:13:51.444948    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="9631b12e-5403-4e99-ac8c-f82116234653" containerName="main"
Jul 16 01:13:51.447948 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:13:51.447926    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-f9f50d/auth-test-model-f9f50d-kserve-769684b9d7-bd4lz"
Jul 16 01:13:51.450415 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:13:51.450395    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-f9f50d\"/\"default-dockercfg-qkwpp\""
Jul 16 01:13:51.450544 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:13:51.450398    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-f9f50d\"/\"kube-root-ca.crt\""
Jul 16 01:13:51.450544 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:13:51.450398    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-f9f50d\"/\"auth-test-model-f9f50d-kserve-self-signed-certs\""
Jul 16 01:13:51.450544 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:13:51.450452    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-f9f50d\"/\"openshift-service-ca.crt\""
Jul 16 01:13:51.461162 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:13:51.461133    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-f9f50d/auth-test-model-f9f50d-kserve-769684b9d7-bd4lz"]
Jul 16 01:13:51.500852 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:13:51.500817    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8102b57a-b19f-4d1a-ad9d-3e4d76588b98-home\") pod \"auth-test-model-f9f50d-kserve-769684b9d7-bd4lz\" (UID: \"8102b57a-b19f-4d1a-ad9d-3e4d76588b98\") " pod="ai-tenant-e2e-shared-a-f9f50d/auth-test-model-f9f50d-kserve-769684b9d7-bd4lz"
Jul 16 01:13:51.500852 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:13:51.500856    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8102b57a-b19f-4d1a-ad9d-3e4d76588b98-tls-certs\") pod \"auth-test-model-f9f50d-kserve-769684b9d7-bd4lz\" (UID: \"8102b57a-b19f-4d1a-ad9d-3e4d76588b98\") " pod="ai-tenant-e2e-shared-a-f9f50d/auth-test-model-f9f50d-kserve-769684b9d7-bd4lz"
Jul 16 01:13:51.501060 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:13:51.500946    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbtmf\" (UniqueName: \"kubernetes.io/projected/8102b57a-b19f-4d1a-ad9d-3e4d76588b98-kube-api-access-bbtmf\") pod \"auth-test-model-f9f50d-kserve-769684b9d7-bd4lz\" (UID: \"8102b57a-b19f-4d1a-ad9d-3e4d76588b98\") " pod="ai-tenant-e2e-shared-a-f9f50d/auth-test-model-f9f50d-kserve-769684b9d7-bd4lz"
Jul 16 01:13:51.501060 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:13:51.501024    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8102b57a-b19f-4d1a-ad9d-3e4d76588b98-tmp-dir\") pod \"auth-test-model-f9f50d-kserve-769684b9d7-bd4lz\" (UID: \"8102b57a-b19f-4d1a-ad9d-3e4d76588b98\") " pod="ai-tenant-e2e-shared-a-f9f50d/auth-test-model-f9f50d-kserve-769684b9d7-bd4lz"
Jul 16 01:13:51.501130 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:13:51.501078    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8102b57a-b19f-4d1a-ad9d-3e4d76588b98-dshm\") pod \"auth-test-model-f9f50d-kserve-769684b9d7-bd4lz\" (UID: \"8102b57a-b19f-4d1a-ad9d-3e4d76588b98\") " pod="ai-tenant-e2e-shared-a-f9f50d/auth-test-model-f9f50d-kserve-769684b9d7-bd4lz"
Jul 16 01:13:51.501130 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:13:51.501096    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8102b57a-b19f-4d1a-ad9d-3e4d76588b98-model-cache\") pod \"auth-test-model-f9f50d-kserve-769684b9d7-bd4lz\" (UID: \"8102b57a-b19f-4d1a-ad9d-3e4d76588b98\") " pod="ai-tenant-e2e-shared-a-f9f50d/auth-test-model-f9f50d-kserve-769684b9d7-bd4lz"
Jul 16 01:13:51.601819 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:13:51.601781    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8102b57a-b19f-4d1a-ad9d-3e4d76588b98-tmp-dir\") pod \"auth-test-model-f9f50d-kserve-769684b9d7-bd4lz\" (UID: \"8102b57a-b19f-4d1a-ad9d-3e4d76588b98\") " pod="ai-tenant-e2e-shared-a-f9f50d/auth-test-model-f9f50d-kserve-769684b9d7-bd4lz"
Jul 16 01:13:51.602004 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:13:51.601847    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8102b57a-b19f-4d1a-ad9d-3e4d76588b98-dshm\") pod \"auth-test-model-f9f50d-kserve-769684b9d7-bd4lz\" (UID: \"8102b57a-b19f-4d1a-ad9d-3e4d76588b98\") " pod="ai-tenant-e2e-shared-a-f9f50d/auth-test-model-f9f50d-kserve-769684b9d7-bd4lz"
Jul 16 01:13:51.602004 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:13:51.601889    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8102b57a-b19f-4d1a-ad9d-3e4d76588b98-model-cache\") pod \"auth-test-model-f9f50d-kserve-769684b9d7-bd4lz\" (UID: \"8102b57a-b19f-4d1a-ad9d-3e4d76588b98\") " pod="ai-tenant-e2e-shared-a-f9f50d/auth-test-model-f9f50d-kserve-769684b9d7-bd4lz"
Jul 16 01:13:51.602004 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:13:51.601940    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8102b57a-b19f-4d1a-ad9d-3e4d76588b98-home\") pod \"auth-test-model-f9f50d-kserve-769684b9d7-bd4lz\" (UID: \"8102b57a-b19f-4d1a-ad9d-3e4d76588b98\") " pod="ai-tenant-e2e-shared-a-f9f50d/auth-test-model-f9f50d-kserve-769684b9d7-bd4lz"
Jul 16 01:13:51.602004 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:13:51.601957    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8102b57a-b19f-4d1a-ad9d-3e4d76588b98-tls-certs\") pod \"auth-test-model-f9f50d-kserve-769684b9d7-bd4lz\" (UID: \"8102b57a-b19f-4d1a-ad9d-3e4d76588b98\") " pod="ai-tenant-e2e-shared-a-f9f50d/auth-test-model-f9f50d-kserve-769684b9d7-bd4lz"
Jul 16 01:13:51.602004 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:13:51.601990    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbtmf\" (UniqueName: \"kubernetes.io/projected/8102b57a-b19f-4d1a-ad9d-3e4d76588b98-kube-api-access-bbtmf\") pod \"auth-test-model-f9f50d-kserve-769684b9d7-bd4lz\" (UID: \"8102b57a-b19f-4d1a-ad9d-3e4d76588b98\") " pod="ai-tenant-e2e-shared-a-f9f50d/auth-test-model-f9f50d-kserve-769684b9d7-bd4lz"
Jul 16 01:13:51.602304 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:13:51.602280    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8102b57a-b19f-4d1a-ad9d-3e4d76588b98-tmp-dir\") pod \"auth-test-model-f9f50d-kserve-769684b9d7-bd4lz\" (UID: \"8102b57a-b19f-4d1a-ad9d-3e4d76588b98\") " pod="ai-tenant-e2e-shared-a-f9f50d/auth-test-model-f9f50d-kserve-769684b9d7-bd4lz"
Jul 16 01:13:51.602304 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:13:51.602296    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8102b57a-b19f-4d1a-ad9d-3e4d76588b98-model-cache\") pod \"auth-test-model-f9f50d-kserve-769684b9d7-bd4lz\" (UID: \"8102b57a-b19f-4d1a-ad9d-3e4d76588b98\") " pod="ai-tenant-e2e-shared-a-f9f50d/auth-test-model-f9f50d-kserve-769684b9d7-bd4lz"
Jul 16 01:13:51.602409 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:13:51.602395    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8102b57a-b19f-4d1a-ad9d-3e4d76588b98-home\") pod \"auth-test-model-f9f50d-kserve-769684b9d7-bd4lz\" (UID: \"8102b57a-b19f-4d1a-ad9d-3e4d76588b98\") " pod="ai-tenant-e2e-shared-a-f9f50d/auth-test-model-f9f50d-kserve-769684b9d7-bd4lz"
Jul 16 01:13:51.604358 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:13:51.604333    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8102b57a-b19f-4d1a-ad9d-3e4d76588b98-dshm\") pod \"auth-test-model-f9f50d-kserve-769684b9d7-bd4lz\" (UID: \"8102b57a-b19f-4d1a-ad9d-3e4d76588b98\") " pod="ai-tenant-e2e-shared-a-f9f50d/auth-test-model-f9f50d-kserve-769684b9d7-bd4lz"
Jul 16 01:13:51.604550 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:13:51.604523    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8102b57a-b19f-4d1a-ad9d-3e4d76588b98-tls-certs\") pod \"auth-test-model-f9f50d-kserve-769684b9d7-bd4lz\" (UID: \"8102b57a-b19f-4d1a-ad9d-3e4d76588b98\") " pod="ai-tenant-e2e-shared-a-f9f50d/auth-test-model-f9f50d-kserve-769684b9d7-bd4lz"
Jul 16 01:13:51.614217 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:13:51.614195    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbtmf\" (UniqueName: \"kubernetes.io/projected/8102b57a-b19f-4d1a-ad9d-3e4d76588b98-kube-api-access-bbtmf\") pod \"auth-test-model-f9f50d-kserve-769684b9d7-bd4lz\" (UID: \"8102b57a-b19f-4d1a-ad9d-3e4d76588b98\") " pod="ai-tenant-e2e-shared-a-f9f50d/auth-test-model-f9f50d-kserve-769684b9d7-bd4lz"
Jul 16 01:13:51.758273 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:13:51.758171    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-f9f50d/auth-test-model-f9f50d-kserve-769684b9d7-bd4lz"
Jul 16 01:13:51.888445 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:13:51.888418    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-f9f50d/auth-test-model-f9f50d-kserve-769684b9d7-bd4lz"]
Jul 16 01:13:51.889969 ip-10-0-143-65 kubenswrapper[2563]: W0716 01:13:51.889944    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8102b57a_b19f_4d1a_ad9d_3e4d76588b98.slice/crio-7ff354753606417b7b73420cc09b004635043c315647e496ce7dae44e3a5bfa2 WatchSource:0}: Error finding container 7ff354753606417b7b73420cc09b004635043c315647e496ce7dae44e3a5bfa2: Status 404 returned error can't find the container with id 7ff354753606417b7b73420cc09b004635043c315647e496ce7dae44e3a5bfa2
Jul 16 01:13:52.037211 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:13:52.037117    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-f9f50d/auth-test-model-f9f50d-kserve-769684b9d7-bd4lz" event={"ID":"8102b57a-b19f-4d1a-ad9d-3e4d76588b98","Type":"ContainerStarted","Data":"bdb316ccff3b6847660d39e242a413c6ccd4227e2dcf41b9f7aeac9d32fb2ef7"}
Jul 16 01:13:52.037211 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:13:52.037168    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-f9f50d/auth-test-model-f9f50d-kserve-769684b9d7-bd4lz" event={"ID":"8102b57a-b19f-4d1a-ad9d-3e4d76588b98","Type":"ContainerStarted","Data":"7ff354753606417b7b73420cc09b004635043c315647e496ce7dae44e3a5bfa2"}
Jul 16 01:13:52.057277 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:13:52.057224    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-f9f50d/auth-test-model-f9f50d-kserve-769684b9d7-bd4lz" podStartSLOduration=1.057207708 podStartE2EDuration="1.057207708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 01:13:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 01:13:52.056368484 +0000 UTC m=+2004.491630224" watchObservedRunningTime="2026-07-16 01:13:52.057207708 +0000 UTC m=+2004.492469460"
Jul 16 01:13:58.568347 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:13:58.568307    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl7fx"]
Jul 16 01:14:01.758307 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:14:01.758267    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-f9f50d/auth-test-model-f9f50d-kserve-769684b9d7-bd4lz"
Jul 16 01:14:01.758736 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:14:01.758354    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-f9f50d/auth-test-model-f9f50d-kserve-769684b9d7-bd4lz"
Jul 16 01:14:01.770979 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:14:01.770944    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-f9f50d/auth-test-model-f9f50d-kserve-769684b9d7-bd4lz"
Jul 16 01:14:02.093264 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:14:02.093234    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-f9f50d/auth-test-model-f9f50d-kserve-769684b9d7-bd4lz"
Jul 16 01:14:09.349487 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:14:09.349451    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-628c71/auth-test-model-628c71-kserve-c4fb454d7-vf9xr"]
Jul 16 01:14:09.352759 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:14:09.352742    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-628c71/auth-test-model-628c71-kserve-c4fb454d7-vf9xr"
Jul 16 01:14:09.354841 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:14:09.354815    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-628c71\"/\"kube-root-ca.crt\""
Jul 16 01:14:09.355679 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:14:09.355662    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-628c71\"/\"openshift-service-ca.crt\""
Jul 16 01:14:09.355778 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:14:09.355693    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-628c71\"/\"default-dockercfg-f9hx6\""
Jul 16 01:14:09.355778 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:14:09.355739    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-628c71\"/\"auth-test-model-628c71-kserve-self-signed-certs\""
Jul 16 01:14:09.361996 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:14:09.361974    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-628c71/auth-test-model-628c71-kserve-c4fb454d7-vf9xr"]
Jul 16 01:14:09.471444 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:14:09.471409    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdgqf\" (UniqueName: \"kubernetes.io/projected/206a9110-d02d-4b2e-93a2-db6998cf5d7d-kube-api-access-bdgqf\") pod \"auth-test-model-628c71-kserve-c4fb454d7-vf9xr\" (UID: \"206a9110-d02d-4b2e-93a2-db6998cf5d7d\") " pod="ai-tenant-e2e-shared-b-628c71/auth-test-model-628c71-kserve-c4fb454d7-vf9xr"
Jul 16 01:14:09.471588 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:14:09.471451    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/206a9110-d02d-4b2e-93a2-db6998cf5d7d-tmp-dir\") pod \"auth-test-model-628c71-kserve-c4fb454d7-vf9xr\" (UID: \"206a9110-d02d-4b2e-93a2-db6998cf5d7d\") " pod="ai-tenant-e2e-shared-b-628c71/auth-test-model-628c71-kserve-c4fb454d7-vf9xr"
Jul 16 01:14:09.471588 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:14:09.471525    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/206a9110-d02d-4b2e-93a2-db6998cf5d7d-model-cache\") pod \"auth-test-model-628c71-kserve-c4fb454d7-vf9xr\" (UID: \"206a9110-d02d-4b2e-93a2-db6998cf5d7d\") " pod="ai-tenant-e2e-shared-b-628c71/auth-test-model-628c71-kserve-c4fb454d7-vf9xr"
Jul 16 01:14:09.471693 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:14:09.471588    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/206a9110-d02d-4b2e-93a2-db6998cf5d7d-home\") pod \"auth-test-model-628c71-kserve-c4fb454d7-vf9xr\" (UID: \"206a9110-d02d-4b2e-93a2-db6998cf5d7d\") " pod="ai-tenant-e2e-shared-b-628c71/auth-test-model-628c71-kserve-c4fb454d7-vf9xr"
Jul 16 01:14:09.471693 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:14:09.471618    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/206a9110-d02d-4b2e-93a2-db6998cf5d7d-tls-certs\") pod \"auth-test-model-628c71-kserve-c4fb454d7-vf9xr\" (UID: \"206a9110-d02d-4b2e-93a2-db6998cf5d7d\") " pod="ai-tenant-e2e-shared-b-628c71/auth-test-model-628c71-kserve-c4fb454d7-vf9xr"
Jul 16 01:14:09.471693 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:14:09.471657    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/206a9110-d02d-4b2e-93a2-db6998cf5d7d-dshm\") pod \"auth-test-model-628c71-kserve-c4fb454d7-vf9xr\" (UID: \"206a9110-d02d-4b2e-93a2-db6998cf5d7d\") " pod="ai-tenant-e2e-shared-b-628c71/auth-test-model-628c71-kserve-c4fb454d7-vf9xr"
Jul 16 01:14:09.572958 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:14:09.572907    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/206a9110-d02d-4b2e-93a2-db6998cf5d7d-tmp-dir\") pod \"auth-test-model-628c71-kserve-c4fb454d7-vf9xr\" (UID: \"206a9110-d02d-4b2e-93a2-db6998cf5d7d\") " pod="ai-tenant-e2e-shared-b-628c71/auth-test-model-628c71-kserve-c4fb454d7-vf9xr"
Jul 16 01:14:09.573155 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:14:09.572974    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/206a9110-d02d-4b2e-93a2-db6998cf5d7d-model-cache\") pod \"auth-test-model-628c71-kserve-c4fb454d7-vf9xr\" (UID: \"206a9110-d02d-4b2e-93a2-db6998cf5d7d\") " pod="ai-tenant-e2e-shared-b-628c71/auth-test-model-628c71-kserve-c4fb454d7-vf9xr"
Jul 16 01:14:09.573155 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:14:09.573006    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/206a9110-d02d-4b2e-93a2-db6998cf5d7d-home\") pod \"auth-test-model-628c71-kserve-c4fb454d7-vf9xr\" (UID: \"206a9110-d02d-4b2e-93a2-db6998cf5d7d\") " pod="ai-tenant-e2e-shared-b-628c71/auth-test-model-628c71-kserve-c4fb454d7-vf9xr"
Jul 16 01:14:09.573155 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:14:09.573038    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/206a9110-d02d-4b2e-93a2-db6998cf5d7d-tls-certs\") pod \"auth-test-model-628c71-kserve-c4fb454d7-vf9xr\" (UID: \"206a9110-d02d-4b2e-93a2-db6998cf5d7d\") " pod="ai-tenant-e2e-shared-b-628c71/auth-test-model-628c71-kserve-c4fb454d7-vf9xr"
Jul 16 01:14:09.573155 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:14:09.573084    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/206a9110-d02d-4b2e-93a2-db6998cf5d7d-dshm\") pod \"auth-test-model-628c71-kserve-c4fb454d7-vf9xr\" (UID: \"206a9110-d02d-4b2e-93a2-db6998cf5d7d\") " pod="ai-tenant-e2e-shared-b-628c71/auth-test-model-628c71-kserve-c4fb454d7-vf9xr"
Jul 16 01:14:09.573155 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:14:09.573150    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdgqf\" (UniqueName: \"kubernetes.io/projected/206a9110-d02d-4b2e-93a2-db6998cf5d7d-kube-api-access-bdgqf\") pod \"auth-test-model-628c71-kserve-c4fb454d7-vf9xr\" (UID: \"206a9110-d02d-4b2e-93a2-db6998cf5d7d\") " pod="ai-tenant-e2e-shared-b-628c71/auth-test-model-628c71-kserve-c4fb454d7-vf9xr"
Jul 16 01:14:09.573392 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:14:09.573372    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/206a9110-d02d-4b2e-93a2-db6998cf5d7d-tmp-dir\") pod \"auth-test-model-628c71-kserve-c4fb454d7-vf9xr\" (UID: \"206a9110-d02d-4b2e-93a2-db6998cf5d7d\") " pod="ai-tenant-e2e-shared-b-628c71/auth-test-model-628c71-kserve-c4fb454d7-vf9xr"
Jul 16 01:14:09.573446 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:14:09.573396    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/206a9110-d02d-4b2e-93a2-db6998cf5d7d-home\") pod \"auth-test-model-628c71-kserve-c4fb454d7-vf9xr\" (UID: \"206a9110-d02d-4b2e-93a2-db6998cf5d7d\") " pod="ai-tenant-e2e-shared-b-628c71/auth-test-model-628c71-kserve-c4fb454d7-vf9xr"
Jul 16 01:14:09.573446 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:14:09.573433    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/206a9110-d02d-4b2e-93a2-db6998cf5d7d-model-cache\") pod \"auth-test-model-628c71-kserve-c4fb454d7-vf9xr\" (UID: \"206a9110-d02d-4b2e-93a2-db6998cf5d7d\") " pod="ai-tenant-e2e-shared-b-628c71/auth-test-model-628c71-kserve-c4fb454d7-vf9xr"
Jul 16 01:14:09.575404 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:14:09.575379    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/206a9110-d02d-4b2e-93a2-db6998cf5d7d-dshm\") pod \"auth-test-model-628c71-kserve-c4fb454d7-vf9xr\" (UID: \"206a9110-d02d-4b2e-93a2-db6998cf5d7d\") " pod="ai-tenant-e2e-shared-b-628c71/auth-test-model-628c71-kserve-c4fb454d7-vf9xr"
Jul 16 01:14:09.575647 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:14:09.575632    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/206a9110-d02d-4b2e-93a2-db6998cf5d7d-tls-certs\") pod \"auth-test-model-628c71-kserve-c4fb454d7-vf9xr\" (UID: \"206a9110-d02d-4b2e-93a2-db6998cf5d7d\") " pod="ai-tenant-e2e-shared-b-628c71/auth-test-model-628c71-kserve-c4fb454d7-vf9xr"
Jul 16 01:14:09.581254 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:14:09.581235    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdgqf\" (UniqueName: \"kubernetes.io/projected/206a9110-d02d-4b2e-93a2-db6998cf5d7d-kube-api-access-bdgqf\") pod \"auth-test-model-628c71-kserve-c4fb454d7-vf9xr\" (UID: \"206a9110-d02d-4b2e-93a2-db6998cf5d7d\") " pod="ai-tenant-e2e-shared-b-628c71/auth-test-model-628c71-kserve-c4fb454d7-vf9xr"
Jul 16 01:14:09.663720 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:14:09.663629    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-628c71/auth-test-model-628c71-kserve-c4fb454d7-vf9xr"
Jul 16 01:14:09.793581 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:14:09.793547    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-628c71/auth-test-model-628c71-kserve-c4fb454d7-vf9xr"]
Jul 16 01:14:09.796303 ip-10-0-143-65 kubenswrapper[2563]: W0716 01:14:09.796275    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod206a9110_d02d_4b2e_93a2_db6998cf5d7d.slice/crio-0b07c7d76d5d955d937035cd7c68c50892c0d9b651d74d81825b48fb6a2d467f WatchSource:0}: Error finding container 0b07c7d76d5d955d937035cd7c68c50892c0d9b651d74d81825b48fb6a2d467f: Status 404 returned error can't find the container with id 0b07c7d76d5d955d937035cd7c68c50892c0d9b651d74d81825b48fb6a2d467f
Jul 16 01:14:10.126240 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:14:10.126206    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-628c71/auth-test-model-628c71-kserve-c4fb454d7-vf9xr" event={"ID":"206a9110-d02d-4b2e-93a2-db6998cf5d7d","Type":"ContainerStarted","Data":"a2050ff512fceba3dbe0fa3506d246872ca8f794cadc421580901d63b6f7cacc"}
Jul 16 01:14:10.126404 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:14:10.126248    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-628c71/auth-test-model-628c71-kserve-c4fb454d7-vf9xr" event={"ID":"206a9110-d02d-4b2e-93a2-db6998cf5d7d","Type":"ContainerStarted","Data":"0b07c7d76d5d955d937035cd7c68c50892c0d9b651d74d81825b48fb6a2d467f"}
Jul 16 01:14:10.144180 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:14:10.144137    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-628c71/auth-test-model-628c71-kserve-c4fb454d7-vf9xr" podStartSLOduration=1.144123203 podStartE2EDuration="1.144123203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 01:14:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 01:14:10.141902307 +0000 UTC m=+2022.577164045" watchObservedRunningTime="2026-07-16 01:14:10.144123203 +0000 UTC m=+2022.579384941"
Jul 16 01:14:14.656287 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:14:14.656249    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl7fx"]
Jul 16 01:14:19.664706 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:14:19.664669    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-628c71/auth-test-model-628c71-kserve-c4fb454d7-vf9xr"
Jul 16 01:14:19.665339 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:14:19.664721    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-628c71/auth-test-model-628c71-kserve-c4fb454d7-vf9xr"
Jul 16 01:14:19.677390 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:14:19.677369    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-628c71/auth-test-model-628c71-kserve-c4fb454d7-vf9xr"
Jul 16 01:14:20.181924 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:14:20.181896    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-628c71/auth-test-model-628c71-kserve-c4fb454d7-vf9xr"
Jul 16 01:14:27.340132 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:14:27.340095    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl7fx"]
Jul 16 01:14:49.049359 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:14:49.049322    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl7fx"]
Jul 16 01:15:00.153447 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:15:00.153332    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7d5e85-29736075-cvfwd"]
Jul 16 01:15:00.156905 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:15:00.156886    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7d5e85-29736075-cvfwd"
Jul 16 01:15:00.159283 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:15:00.159267    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-trkf6\""
Jul 16 01:15:00.166032 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:15:00.166011    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7d5e85-29736075-cvfwd"
Jul 16 01:15:00.183716 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:15:00.183683    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7d5e85-29736075-cvfwd"]
Jul 16 01:15:00.205028 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:15:00.204995    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-2814a2-29736075-8sw6g"]
Jul 16 01:15:00.209070 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:15:00.209049    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29736075-qhm25"]
Jul 16 01:15:00.209182 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:15:00.209172    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-2814a2-29736075-8sw6g"
Jul 16 01:15:00.212291 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:15:00.212223    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29736075-qhm25"
Jul 16 01:15:00.229984 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:15:00.225556    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29736075-qhm25"]
Jul 16 01:15:00.229984 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:15:00.225688    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-2814a2-29736075-8sw6g"
Jul 16 01:15:00.229984 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:15:00.228456    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-2814a2-29736075-8sw6g"]
Jul 16 01:15:00.235918 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:15:00.235639    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29736075-qhm25"
Jul 16 01:15:00.340605 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:15:00.340578    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7d5e85-29736075-cvfwd"]
Jul 16 01:15:00.342820 ip-10-0-143-65 kubenswrapper[2563]: W0716 01:15:00.342769    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61ee0ce1_2f13_407a_a3c5_9f67846606dc.slice/crio-17ca78266b681b272bcbadb9a53112f8a0b284d0dbbc6203c8cbe844d3cc582a WatchSource:0}: Error finding container 17ca78266b681b272bcbadb9a53112f8a0b284d0dbbc6203c8cbe844d3cc582a: Status 404 returned error can't find the container with id 17ca78266b681b272bcbadb9a53112f8a0b284d0dbbc6203c8cbe844d3cc582a
Jul 16 01:15:00.345855 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:15:00.345830    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 16 01:15:00.364735 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:15:00.364687    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7d5e85-29736075-cvfwd" event={"ID":"61ee0ce1-2f13-407a-a3c5-9f67846606dc","Type":"ContainerStarted","Data":"17ca78266b681b272bcbadb9a53112f8a0b284d0dbbc6203c8cbe844d3cc582a"}
Jul 16 01:15:00.418179 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:15:00.418137    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29736075-qhm25"]
Jul 16 01:15:00.420447 ip-10-0-143-65 kubenswrapper[2563]: W0716 01:15:00.420419    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1658112c_ac54_462f_badd_ebd7d9c24d14.slice/crio-837d149e82d22a72f6cd2f31e2a64d7b4c0b139793ed54aad65571281ca4081e WatchSource:0}: Error finding container 837d149e82d22a72f6cd2f31e2a64d7b4c0b139793ed54aad65571281ca4081e: Status 404 returned error can't find the container with id 837d149e82d22a72f6cd2f31e2a64d7b4c0b139793ed54aad65571281ca4081e
Jul 16 01:15:00.616995 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:15:00.616853    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-2814a2-29736075-8sw6g"]
Jul 16 01:15:00.618942 ip-10-0-143-65 kubenswrapper[2563]: W0716 01:15:00.618920    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode494d6b2_80c8_42fb_8678_e40796989e48.slice/crio-17db2a62f1278e2e26547648bf86341e051cd622947e46e7fbdd6a9884f78398 WatchSource:0}: Error finding container 17db2a62f1278e2e26547648bf86341e051cd622947e46e7fbdd6a9884f78398: Status 404 returned error can't find the container with id 17db2a62f1278e2e26547648bf86341e051cd622947e46e7fbdd6a9884f78398
Jul 16 01:15:01.371670 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:15:01.371633    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-2814a2-29736075-8sw6g" event={"ID":"e494d6b2-80c8-42fb-8678-e40796989e48","Type":"ContainerStarted","Data":"17db2a62f1278e2e26547648bf86341e051cd622947e46e7fbdd6a9884f78398"}
Jul 16 01:15:01.373229 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:15:01.373183    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29736075-qhm25" event={"ID":"1658112c-ac54-462f-badd-ebd7d9c24d14","Type":"ContainerStarted","Data":"837d149e82d22a72f6cd2f31e2a64d7b4c0b139793ed54aad65571281ca4081e"}
Jul 16 01:15:03.384949 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:15:03.384837    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7d5e85-29736075-cvfwd" event={"ID":"61ee0ce1-2f13-407a-a3c5-9f67846606dc","Type":"ContainerStarted","Data":"2e0e021729baf02bc1292a53b38b9a4a8993aa064e97268beeee92bd630252ed"}
Jul 16 01:15:03.386397 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:15:03.386369    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29736075-qhm25" event={"ID":"1658112c-ac54-462f-badd-ebd7d9c24d14","Type":"ContainerStarted","Data":"9dab2357ea31d02f162fa81ae2f0299edead5701e0d94adad5186799474c773e"}
Jul 16 01:15:03.387736 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:15:03.387713    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-2814a2-29736075-8sw6g" event={"ID":"e494d6b2-80c8-42fb-8678-e40796989e48","Type":"ContainerStarted","Data":"047a6282eff75f2d62f55aa4b74f2a80fae68d5e276be9c1b1ae35b62c23124c"}
Jul 16 01:15:03.412280 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:15:03.412232    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7d5e85-29736075-cvfwd" podStartSLOduration=0.882585719 podStartE2EDuration="3.412216844s" totalImagesPullingTime="2.529631125s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 01:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 01:15:03.410833543 +0000 UTC m=+2075.846095272" watchObservedRunningTime="2026-07-16 01:15:03.412216844 +0000 UTC m=+2075.847478610"
Jul 16 01:15:03.438153 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:15:03.438101    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29736075-qhm25" podStartSLOduration=0.997664815 podStartE2EDuration="3.43808579s" totalImagesPullingTime="2.440420975s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 01:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 01:15:03.437068324 +0000 UTC m=+2075.872330062" watchObservedRunningTime="2026-07-16 01:15:03.43808579 +0000 UTC m=+2075.873347552"
Jul 16 01:15:03.472772 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:15:03.472713    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-2814a2-29736075-8sw6g" podStartSLOduration=1.221792399 podStartE2EDuration="3.472691787s" totalImagesPullingTime="2.250899388s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 01:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 01:15:03.469196108 +0000 UTC m=+2075.904457847" watchObservedRunningTime="2026-07-16 01:15:03.472691787 +0000 UTC m=+2075.907953527"
Jul 16 01:15:04.394527 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:15:04.394493    2563 generic.go:356] "Generic (PLEG): container finished" podID="61ee0ce1-2f13-407a-a3c5-9f67846606dc" containerID="2e0e021729baf02bc1292a53b38b9a4a8993aa064e97268beeee92bd630252ed" exitCode=0
Jul 16 01:15:04.394976 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:15:04.394578    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7d5e85-29736075-cvfwd" event={"ID":"61ee0ce1-2f13-407a-a3c5-9f67846606dc","Type":"ContainerDied","Data":"2e0e021729baf02bc1292a53b38b9a4a8993aa064e97268beeee92bd630252ed"}
Jul 16 01:15:04.396126 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:15:04.396099    2563 generic.go:356] "Generic (PLEG): container finished" podID="1658112c-ac54-462f-badd-ebd7d9c24d14" containerID="9dab2357ea31d02f162fa81ae2f0299edead5701e0d94adad5186799474c773e" exitCode=0
Jul 16 01:15:04.396237 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:15:04.396126    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29736075-qhm25" event={"ID":"1658112c-ac54-462f-badd-ebd7d9c24d14","Type":"ContainerDied","Data":"9dab2357ea31d02f162fa81ae2f0299edead5701e0d94adad5186799474c773e"}
Jul 16 01:15:04.397624 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:15:04.397603    2563 generic.go:356] "Generic (PLEG): container finished" podID="e494d6b2-80c8-42fb-8678-e40796989e48" containerID="047a6282eff75f2d62f55aa4b74f2a80fae68d5e276be9c1b1ae35b62c23124c" exitCode=0
Jul 16 01:15:04.397726 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:15:04.397639    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-2814a2-29736075-8sw6g" event={"ID":"e494d6b2-80c8-42fb-8678-e40796989e48","Type":"ContainerDied","Data":"047a6282eff75f2d62f55aa4b74f2a80fae68d5e276be9c1b1ae35b62c23124c"}
Jul 16 01:15:05.534628 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:15:05.534600    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29736075-qhm25"
Jul 16 01:15:05.581342 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:15:05.581318    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7d5e85-29736075-cvfwd"
Jul 16 01:15:05.596395 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:15:05.596371    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-2814a2-29736075-8sw6g"
Jul 16 01:15:06.247733 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:15:06.247698    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl7fx"]
Jul 16 01:15:06.407638 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:15:06.407609    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-2814a2-29736075-8sw6g"
Jul 16 01:15:06.407818 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:15:06.407601    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-2814a2-29736075-8sw6g" event={"ID":"e494d6b2-80c8-42fb-8678-e40796989e48","Type":"ContainerDied","Data":"17db2a62f1278e2e26547648bf86341e051cd622947e46e7fbdd6a9884f78398"}
Jul 16 01:15:06.407818 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:15:06.407787    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="17db2a62f1278e2e26547648bf86341e051cd622947e46e7fbdd6a9884f78398"
Jul 16 01:15:06.409005 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:15:06.408977    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7d5e85-29736075-cvfwd"
Jul 16 01:15:06.409005 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:15:06.408988    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7d5e85-29736075-cvfwd" event={"ID":"61ee0ce1-2f13-407a-a3c5-9f67846606dc","Type":"ContainerDied","Data":"17ca78266b681b272bcbadb9a53112f8a0b284d0dbbc6203c8cbe844d3cc582a"}
Jul 16 01:15:06.409165 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:15:06.409026    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="17ca78266b681b272bcbadb9a53112f8a0b284d0dbbc6203c8cbe844d3cc582a"
Jul 16 01:15:06.410479 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:15:06.410460    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29736075-qhm25" event={"ID":"1658112c-ac54-462f-badd-ebd7d9c24d14","Type":"ContainerDied","Data":"837d149e82d22a72f6cd2f31e2a64d7b4c0b139793ed54aad65571281ca4081e"}
Jul 16 01:15:06.410596 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:15:06.410499    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="837d149e82d22a72f6cd2f31e2a64d7b4c0b139793ed54aad65571281ca4081e"
Jul 16 01:15:06.410596 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:15:06.410469    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29736075-qhm25"
Jul 16 01:15:23.458738 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:15:23.458700    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl7fx"]
Jul 16 01:15:28.041303 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:15:28.041276    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bn5hr_e990a21d-614b-4d1d-be54-0df6e44487ed/ovn-acl-logging/0.log"
Jul 16 01:15:28.048091 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:15:28.048069    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bn5hr_e990a21d-614b-4d1d-be54-0df6e44487ed/ovn-acl-logging/0.log"
Jul 16 01:15:44.643272 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:15:44.643231    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl7fx"]
Jul 16 01:16:06.352584 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:16:06.352527    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl7fx"]
Jul 16 01:16:27.345527 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:16:27.345448    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl7fx"]
Jul 16 01:16:44.054091 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:16:44.054053    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-f9f50d/auth-test-model-f9f50d-kserve-769684b9d7-bd4lz"]
Jul 16 01:16:45.077666 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:16:45.077634    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-628c71/auth-test-model-628c71-kserve-c4fb454d7-vf9xr"]
Jul 16 01:16:47.758927 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:16:47.758888    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl7fx"]
Jul 16 01:16:59.074493 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:16:59.074428    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-f9f50d/auth-test-model-f9f50d-kserve-769684b9d7-bd4lz" podUID="8102b57a-b19f-4d1a-ad9d-3e4d76588b98" containerName="main" containerID="cri-o://bdb316ccff3b6847660d39e242a413c6ccd4227e2dcf41b9f7aeac9d32fb2ef7" gracePeriod=45
Jul 16 01:16:59.325336 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:16:59.325277    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-f9f50d/auth-test-model-f9f50d-kserve-769684b9d7-bd4lz"
Jul 16 01:16:59.401846 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:16:59.401816    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8102b57a-b19f-4d1a-ad9d-3e4d76588b98-dshm\") pod \"8102b57a-b19f-4d1a-ad9d-3e4d76588b98\" (UID: \"8102b57a-b19f-4d1a-ad9d-3e4d76588b98\") "
Jul 16 01:16:59.402040 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:16:59.401903    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8102b57a-b19f-4d1a-ad9d-3e4d76588b98-home\") pod \"8102b57a-b19f-4d1a-ad9d-3e4d76588b98\" (UID: \"8102b57a-b19f-4d1a-ad9d-3e4d76588b98\") "
Jul 16 01:16:59.402040 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:16:59.401938    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bbtmf\" (UniqueName: \"kubernetes.io/projected/8102b57a-b19f-4d1a-ad9d-3e4d76588b98-kube-api-access-bbtmf\") pod \"8102b57a-b19f-4d1a-ad9d-3e4d76588b98\" (UID: \"8102b57a-b19f-4d1a-ad9d-3e4d76588b98\") "
Jul 16 01:16:59.402040 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:16:59.401973    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8102b57a-b19f-4d1a-ad9d-3e4d76588b98-tmp-dir\") pod \"8102b57a-b19f-4d1a-ad9d-3e4d76588b98\" (UID: \"8102b57a-b19f-4d1a-ad9d-3e4d76588b98\") "
Jul 16 01:16:59.402040 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:16:59.402027    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8102b57a-b19f-4d1a-ad9d-3e4d76588b98-model-cache\") pod \"8102b57a-b19f-4d1a-ad9d-3e4d76588b98\" (UID: \"8102b57a-b19f-4d1a-ad9d-3e4d76588b98\") "
Jul 16 01:16:59.402246 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:16:59.402097    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8102b57a-b19f-4d1a-ad9d-3e4d76588b98-tls-certs\") pod \"8102b57a-b19f-4d1a-ad9d-3e4d76588b98\" (UID: \"8102b57a-b19f-4d1a-ad9d-3e4d76588b98\") "
Jul 16 01:16:59.402246 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:16:59.402204    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8102b57a-b19f-4d1a-ad9d-3e4d76588b98-home" (OuterVolumeSpecName: "home") pod "8102b57a-b19f-4d1a-ad9d-3e4d76588b98" (UID: "8102b57a-b19f-4d1a-ad9d-3e4d76588b98"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 01:16:59.402355 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:16:59.402323    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8102b57a-b19f-4d1a-ad9d-3e4d76588b98-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8102b57a-b19f-4d1a-ad9d-3e4d76588b98" (UID: "8102b57a-b19f-4d1a-ad9d-3e4d76588b98"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 01:16:59.402355 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:16:59.402341    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8102b57a-b19f-4d1a-ad9d-3e4d76588b98-home\") on node \"ip-10-0-143-65.ec2.internal\" DevicePath \"\""
Jul 16 01:16:59.402464 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:16:59.402412    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8102b57a-b19f-4d1a-ad9d-3e4d76588b98-model-cache" (OuterVolumeSpecName: "model-cache") pod "8102b57a-b19f-4d1a-ad9d-3e4d76588b98" (UID: "8102b57a-b19f-4d1a-ad9d-3e4d76588b98"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 01:16:59.404240 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:16:59.404202    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8102b57a-b19f-4d1a-ad9d-3e4d76588b98-dshm" (OuterVolumeSpecName: "dshm") pod "8102b57a-b19f-4d1a-ad9d-3e4d76588b98" (UID: "8102b57a-b19f-4d1a-ad9d-3e4d76588b98"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 01:16:59.404378 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:16:59.404278    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8102b57a-b19f-4d1a-ad9d-3e4d76588b98-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8102b57a-b19f-4d1a-ad9d-3e4d76588b98" (UID: "8102b57a-b19f-4d1a-ad9d-3e4d76588b98"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 01:16:59.404378 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:16:59.404288    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8102b57a-b19f-4d1a-ad9d-3e4d76588b98-kube-api-access-bbtmf" (OuterVolumeSpecName: "kube-api-access-bbtmf") pod "8102b57a-b19f-4d1a-ad9d-3e4d76588b98" (UID: "8102b57a-b19f-4d1a-ad9d-3e4d76588b98"). InnerVolumeSpecName "kube-api-access-bbtmf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 01:16:59.503764 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:16:59.503722    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbtmf\" (UniqueName: \"kubernetes.io/projected/8102b57a-b19f-4d1a-ad9d-3e4d76588b98-kube-api-access-bbtmf\") on node \"ip-10-0-143-65.ec2.internal\" DevicePath \"\""
Jul 16 01:16:59.503962 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:16:59.503784    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8102b57a-b19f-4d1a-ad9d-3e4d76588b98-tmp-dir\") on node \"ip-10-0-143-65.ec2.internal\" DevicePath \"\""
Jul 16 01:16:59.503962 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:16:59.503796    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8102b57a-b19f-4d1a-ad9d-3e4d76588b98-model-cache\") on node \"ip-10-0-143-65.ec2.internal\" DevicePath \"\""
Jul 16 01:16:59.503962 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:16:59.503805    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8102b57a-b19f-4d1a-ad9d-3e4d76588b98-tls-certs\") on node \"ip-10-0-143-65.ec2.internal\" DevicePath \"\""
Jul 16 01:16:59.503962 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:16:59.503813    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8102b57a-b19f-4d1a-ad9d-3e4d76588b98-dshm\") on node \"ip-10-0-143-65.ec2.internal\" DevicePath \"\""
Jul 16 01:16:59.954722 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:16:59.954691    2563 generic.go:356] "Generic (PLEG): container finished" podID="8102b57a-b19f-4d1a-ad9d-3e4d76588b98" containerID="bdb316ccff3b6847660d39e242a413c6ccd4227e2dcf41b9f7aeac9d32fb2ef7" exitCode=0
Jul 16 01:16:59.954937 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:16:59.954738    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-f9f50d/auth-test-model-f9f50d-kserve-769684b9d7-bd4lz"
Jul 16 01:16:59.954937 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:16:59.954777    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-f9f50d/auth-test-model-f9f50d-kserve-769684b9d7-bd4lz" event={"ID":"8102b57a-b19f-4d1a-ad9d-3e4d76588b98","Type":"ContainerDied","Data":"bdb316ccff3b6847660d39e242a413c6ccd4227e2dcf41b9f7aeac9d32fb2ef7"}
Jul 16 01:16:59.954937 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:16:59.954821    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-f9f50d/auth-test-model-f9f50d-kserve-769684b9d7-bd4lz" event={"ID":"8102b57a-b19f-4d1a-ad9d-3e4d76588b98","Type":"ContainerDied","Data":"7ff354753606417b7b73420cc09b004635043c315647e496ce7dae44e3a5bfa2"}
Jul 16 01:16:59.954937 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:16:59.954840    2563 scope.go:117] "RemoveContainer" containerID="bdb316ccff3b6847660d39e242a413c6ccd4227e2dcf41b9f7aeac9d32fb2ef7"
Jul 16 01:16:59.964283 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:16:59.964259    2563 scope.go:117] "RemoveContainer" containerID="bdb316ccff3b6847660d39e242a413c6ccd4227e2dcf41b9f7aeac9d32fb2ef7"
Jul 16 01:16:59.964585 ip-10-0-143-65 kubenswrapper[2563]: E0716 01:16:59.964564    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bdb316ccff3b6847660d39e242a413c6ccd4227e2dcf41b9f7aeac9d32fb2ef7\": container with ID starting with bdb316ccff3b6847660d39e242a413c6ccd4227e2dcf41b9f7aeac9d32fb2ef7 not found: ID does not exist" containerID="bdb316ccff3b6847660d39e242a413c6ccd4227e2dcf41b9f7aeac9d32fb2ef7"
Jul 16 01:16:59.964648 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:16:59.964594    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdb316ccff3b6847660d39e242a413c6ccd4227e2dcf41b9f7aeac9d32fb2ef7"} err="failed to get container status \"bdb316ccff3b6847660d39e242a413c6ccd4227e2dcf41b9f7aeac9d32fb2ef7\": rpc error: code = NotFound desc = could not find container \"bdb316ccff3b6847660d39e242a413c6ccd4227e2dcf41b9f7aeac9d32fb2ef7\": container with ID starting with bdb316ccff3b6847660d39e242a413c6ccd4227e2dcf41b9f7aeac9d32fb2ef7 not found: ID does not exist"
Jul 16 01:16:59.974441 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:16:59.974404    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-f9f50d/auth-test-model-f9f50d-kserve-769684b9d7-bd4lz"]
Jul 16 01:16:59.978339 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:16:59.978311    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-f9f50d/auth-test-model-f9f50d-kserve-769684b9d7-bd4lz"]
Jul 16 01:17:00.007113 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:17:00.007067    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8102b57a-b19f-4d1a-ad9d-3e4d76588b98" path="/var/lib/kubelet/pods/8102b57a-b19f-4d1a-ad9d-3e4d76588b98/volumes"
Jul 16 01:17:00.097399 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:17:00.097341    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-628c71/auth-test-model-628c71-kserve-c4fb454d7-vf9xr" podUID="206a9110-d02d-4b2e-93a2-db6998cf5d7d" containerName="main" containerID="cri-o://a2050ff512fceba3dbe0fa3506d246872ca8f794cadc421580901d63b6f7cacc" gracePeriod=45
Jul 16 01:17:00.171661 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:17:00.171613    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-628c71/auth-test-model-628c71-kserve-c4fb454d7-vf9xr" podUID="206a9110-d02d-4b2e-93a2-db6998cf5d7d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": dial tcp 10.134.0.31:8000: connect: connection refused"
Jul 16 01:17:00.359427 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:17:00.359401    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-628c71/auth-test-model-628c71-kserve-c4fb454d7-vf9xr"
Jul 16 01:17:00.412325 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:17:00.412290    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/206a9110-d02d-4b2e-93a2-db6998cf5d7d-dshm\") pod \"206a9110-d02d-4b2e-93a2-db6998cf5d7d\" (UID: \"206a9110-d02d-4b2e-93a2-db6998cf5d7d\") "
Jul 16 01:17:00.412500 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:17:00.412335    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bdgqf\" (UniqueName: \"kubernetes.io/projected/206a9110-d02d-4b2e-93a2-db6998cf5d7d-kube-api-access-bdgqf\") pod \"206a9110-d02d-4b2e-93a2-db6998cf5d7d\" (UID: \"206a9110-d02d-4b2e-93a2-db6998cf5d7d\") "
Jul 16 01:17:00.412500 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:17:00.412400    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/206a9110-d02d-4b2e-93a2-db6998cf5d7d-tls-certs\") pod \"206a9110-d02d-4b2e-93a2-db6998cf5d7d\" (UID: \"206a9110-d02d-4b2e-93a2-db6998cf5d7d\") "
Jul 16 01:17:00.412500 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:17:00.412451    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/206a9110-d02d-4b2e-93a2-db6998cf5d7d-tmp-dir\") pod \"206a9110-d02d-4b2e-93a2-db6998cf5d7d\" (UID: \"206a9110-d02d-4b2e-93a2-db6998cf5d7d\") "
Jul 16 01:17:00.412500 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:17:00.412466    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/206a9110-d02d-4b2e-93a2-db6998cf5d7d-model-cache\") pod \"206a9110-d02d-4b2e-93a2-db6998cf5d7d\" (UID: \"206a9110-d02d-4b2e-93a2-db6998cf5d7d\") "
Jul 16 01:17:00.412500 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:17:00.412488    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/206a9110-d02d-4b2e-93a2-db6998cf5d7d-home\") pod \"206a9110-d02d-4b2e-93a2-db6998cf5d7d\" (UID: \"206a9110-d02d-4b2e-93a2-db6998cf5d7d\") "
Jul 16 01:17:00.412786 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:17:00.412760    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/206a9110-d02d-4b2e-93a2-db6998cf5d7d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "206a9110-d02d-4b2e-93a2-db6998cf5d7d" (UID: "206a9110-d02d-4b2e-93a2-db6998cf5d7d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 01:17:00.412841 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:17:00.412769    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/206a9110-d02d-4b2e-93a2-db6998cf5d7d-model-cache" (OuterVolumeSpecName: "model-cache") pod "206a9110-d02d-4b2e-93a2-db6998cf5d7d" (UID: "206a9110-d02d-4b2e-93a2-db6998cf5d7d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 01:17:00.412841 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:17:00.412812    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/206a9110-d02d-4b2e-93a2-db6998cf5d7d-home" (OuterVolumeSpecName: "home") pod "206a9110-d02d-4b2e-93a2-db6998cf5d7d" (UID: "206a9110-d02d-4b2e-93a2-db6998cf5d7d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 01:17:00.414627 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:17:00.414602    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/206a9110-d02d-4b2e-93a2-db6998cf5d7d-dshm" (OuterVolumeSpecName: "dshm") pod "206a9110-d02d-4b2e-93a2-db6998cf5d7d" (UID: "206a9110-d02d-4b2e-93a2-db6998cf5d7d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 01:17:00.414709 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:17:00.414602    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/206a9110-d02d-4b2e-93a2-db6998cf5d7d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "206a9110-d02d-4b2e-93a2-db6998cf5d7d" (UID: "206a9110-d02d-4b2e-93a2-db6998cf5d7d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 01:17:00.414709 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:17:00.414624    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/206a9110-d02d-4b2e-93a2-db6998cf5d7d-kube-api-access-bdgqf" (OuterVolumeSpecName: "kube-api-access-bdgqf") pod "206a9110-d02d-4b2e-93a2-db6998cf5d7d" (UID: "206a9110-d02d-4b2e-93a2-db6998cf5d7d"). InnerVolumeSpecName "kube-api-access-bdgqf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 01:17:00.513959 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:17:00.513856    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/206a9110-d02d-4b2e-93a2-db6998cf5d7d-tmp-dir\") on node \"ip-10-0-143-65.ec2.internal\" DevicePath \"\""
Jul 16 01:17:00.513959 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:17:00.513904    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/206a9110-d02d-4b2e-93a2-db6998cf5d7d-model-cache\") on node \"ip-10-0-143-65.ec2.internal\" DevicePath \"\""
Jul 16 01:17:00.513959 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:17:00.513913    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/206a9110-d02d-4b2e-93a2-db6998cf5d7d-home\") on node \"ip-10-0-143-65.ec2.internal\" DevicePath \"\""
Jul 16 01:17:00.513959 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:17:00.513922    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/206a9110-d02d-4b2e-93a2-db6998cf5d7d-dshm\") on node \"ip-10-0-143-65.ec2.internal\" DevicePath \"\""
Jul 16 01:17:00.513959 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:17:00.513930    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdgqf\" (UniqueName: \"kubernetes.io/projected/206a9110-d02d-4b2e-93a2-db6998cf5d7d-kube-api-access-bdgqf\") on node \"ip-10-0-143-65.ec2.internal\" DevicePath \"\""
Jul 16 01:17:00.513959 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:17:00.513938    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/206a9110-d02d-4b2e-93a2-db6998cf5d7d-tls-certs\") on node \"ip-10-0-143-65.ec2.internal\" DevicePath \"\""
Jul 16 01:17:00.962121 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:17:00.962085    2563 generic.go:356] "Generic (PLEG): container finished" podID="206a9110-d02d-4b2e-93a2-db6998cf5d7d" containerID="a2050ff512fceba3dbe0fa3506d246872ca8f794cadc421580901d63b6f7cacc" exitCode=0
Jul 16 01:17:00.962121 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:17:00.962124    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-628c71/auth-test-model-628c71-kserve-c4fb454d7-vf9xr" event={"ID":"206a9110-d02d-4b2e-93a2-db6998cf5d7d","Type":"ContainerDied","Data":"a2050ff512fceba3dbe0fa3506d246872ca8f794cadc421580901d63b6f7cacc"}
Jul 16 01:17:00.962386 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:17:00.962134    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-628c71/auth-test-model-628c71-kserve-c4fb454d7-vf9xr"
Jul 16 01:17:00.962386 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:17:00.962147    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-628c71/auth-test-model-628c71-kserve-c4fb454d7-vf9xr" event={"ID":"206a9110-d02d-4b2e-93a2-db6998cf5d7d","Type":"ContainerDied","Data":"0b07c7d76d5d955d937035cd7c68c50892c0d9b651d74d81825b48fb6a2d467f"}
Jul 16 01:17:00.962386 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:17:00.962162    2563 scope.go:117] "RemoveContainer" containerID="a2050ff512fceba3dbe0fa3506d246872ca8f794cadc421580901d63b6f7cacc"
Jul 16 01:17:00.972842 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:17:00.972725    2563 scope.go:117] "RemoveContainer" containerID="a2050ff512fceba3dbe0fa3506d246872ca8f794cadc421580901d63b6f7cacc"
Jul 16 01:17:00.973249 ip-10-0-143-65 kubenswrapper[2563]: E0716 01:17:00.973221    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2050ff512fceba3dbe0fa3506d246872ca8f794cadc421580901d63b6f7cacc\": container with ID starting with a2050ff512fceba3dbe0fa3506d246872ca8f794cadc421580901d63b6f7cacc not found: ID does not exist" containerID="a2050ff512fceba3dbe0fa3506d246872ca8f794cadc421580901d63b6f7cacc"
Jul 16 01:17:00.973341 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:17:00.973259    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2050ff512fceba3dbe0fa3506d246872ca8f794cadc421580901d63b6f7cacc"} err="failed to get container status \"a2050ff512fceba3dbe0fa3506d246872ca8f794cadc421580901d63b6f7cacc\": rpc error: code = NotFound desc = could not find container \"a2050ff512fceba3dbe0fa3506d246872ca8f794cadc421580901d63b6f7cacc\": container with ID starting with a2050ff512fceba3dbe0fa3506d246872ca8f794cadc421580901d63b6f7cacc not found: ID does not exist"
Jul 16 01:17:00.985844 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:17:00.985818    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-628c71/auth-test-model-628c71-kserve-c4fb454d7-vf9xr"]
Jul 16 01:17:00.992493 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:17:00.992458    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-628c71/auth-test-model-628c71-kserve-c4fb454d7-vf9xr"]
Jul 16 01:17:02.005503 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:17:02.005470    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="206a9110-d02d-4b2e-93a2-db6998cf5d7d" path="/var/lib/kubelet/pods/206a9110-d02d-4b2e-93a2-db6998cf5d7d/volumes"
Jul 16 01:17:20.652997 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:17:20.652965    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-f9f50d/sub-test-model-f9f50d-kserve-84b54f49c5-ct8md"]
Jul 16 01:17:20.653362 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:17:20.653309    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e494d6b2-80c8-42fb-8678-e40796989e48" containerName="cleanup"
Jul 16 01:17:20.653362 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:17:20.653321    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e494d6b2-80c8-42fb-8678-e40796989e48" containerName="cleanup"
Jul 16 01:17:20.653362 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:17:20.653337    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61ee0ce1-2f13-407a-a3c5-9f67846606dc" containerName="cleanup"
Jul 16 01:17:20.653362 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:17:20.653344    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="61ee0ce1-2f13-407a-a3c5-9f67846606dc" containerName="cleanup"
Jul 16 01:17:20.653362 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:17:20.653356    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="206a9110-d02d-4b2e-93a2-db6998cf5d7d" containerName="main"
Jul 16 01:17:20.653362 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:17:20.653361    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="206a9110-d02d-4b2e-93a2-db6998cf5d7d" containerName="main"
Jul 16 01:17:20.653550 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:17:20.653374    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1658112c-ac54-462f-badd-ebd7d9c24d14" containerName="cleanup"
Jul 16 01:17:20.653550 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:17:20.653381    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="1658112c-ac54-462f-badd-ebd7d9c24d14" containerName="cleanup"
Jul 16 01:17:20.653550 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:17:20.653392    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8102b57a-b19f-4d1a-ad9d-3e4d76588b98" containerName="main"
Jul 16 01:17:20.653550 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:17:20.653397    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8102b57a-b19f-4d1a-ad9d-3e4d76588b98" containerName="main"
Jul 16 01:17:20.653550 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:17:20.653454    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e494d6b2-80c8-42fb-8678-e40796989e48" containerName="cleanup"
Jul 16 01:17:20.653550 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:17:20.653463    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8102b57a-b19f-4d1a-ad9d-3e4d76588b98" containerName="main"
Jul 16 01:17:20.653550 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:17:20.653470    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="206a9110-d02d-4b2e-93a2-db6998cf5d7d" containerName="main"
Jul 16 01:17:20.653550 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:17:20.653476    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="61ee0ce1-2f13-407a-a3c5-9f67846606dc" containerName="cleanup"
Jul 16 01:17:20.653550 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:17:20.653483    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="1658112c-ac54-462f-badd-ebd7d9c24d14" containerName="cleanup"
Jul 16 01:17:20.656386 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:17:20.656370    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-f9f50d/sub-test-model-f9f50d-kserve-84b54f49c5-ct8md"
Jul 16 01:17:20.658582 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:17:20.658563    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-f9f50d\"/\"kube-root-ca.crt\""
Jul 16 01:17:20.659129 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:17:20.659111    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-f9f50d\"/\"openshift-service-ca.crt\""
Jul 16 01:17:20.659219 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:17:20.659112    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-f9f50d\"/\"default-dockercfg-qkwpp\""
Jul 16 01:17:20.659219 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:17:20.659148    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-f9f50d\"/\"sub-test-model-f9f50d-kserve-self-signed-certs\""
Jul 16 01:17:20.666555 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:17:20.666529    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-f9f50d/sub-test-model-f9f50d-kserve-84b54f49c5-ct8md"]
Jul 16 01:17:20.797885 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:17:20.797829    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4dda17f-9fe1-4aee-a1b8-003c5be77ba0-dshm\") pod \"sub-test-model-f9f50d-kserve-84b54f49c5-ct8md\" (UID: \"f4dda17f-9fe1-4aee-a1b8-003c5be77ba0\") " pod="ai-tenant-e2e-shared-a-f9f50d/sub-test-model-f9f50d-kserve-84b54f49c5-ct8md"
Jul 16 01:17:20.798055 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:17:20.797952    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4dda17f-9fe1-4aee-a1b8-003c5be77ba0-model-cache\") pod \"sub-test-model-f9f50d-kserve-84b54f49c5-ct8md\" (UID: \"f4dda17f-9fe1-4aee-a1b8-003c5be77ba0\") " pod="ai-tenant-e2e-shared-a-f9f50d/sub-test-model-f9f50d-kserve-84b54f49c5-ct8md"
Jul 16 01:17:20.798055 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:17:20.797997    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4dda17f-9fe1-4aee-a1b8-003c5be77ba0-tmp-dir\") pod \"sub-test-model-f9f50d-kserve-84b54f49c5-ct8md\" (UID: \"f4dda17f-9fe1-4aee-a1b8-003c5be77ba0\") " pod="ai-tenant-e2e-shared-a-f9f50d/sub-test-model-f9f50d-kserve-84b54f49c5-ct8md"
Jul 16 01:17:20.798055 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:17:20.798025    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcrrw\" (UniqueName: \"kubernetes.io/projected/f4dda17f-9fe1-4aee-a1b8-003c5be77ba0-kube-api-access-vcrrw\") pod \"sub-test-model-f9f50d-kserve-84b54f49c5-ct8md\" (UID: \"f4dda17f-9fe1-4aee-a1b8-003c5be77ba0\") " pod="ai-tenant-e2e-shared-a-f9f50d/sub-test-model-f9f50d-kserve-84b54f49c5-ct8md"
Jul 16 01:17:20.798161 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:17:20.798106    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4dda17f-9fe1-4aee-a1b8-003c5be77ba0-tls-certs\") pod \"sub-test-model-f9f50d-kserve-84b54f49c5-ct8md\" (UID: \"f4dda17f-9fe1-4aee-a1b8-003c5be77ba0\") " pod="ai-tenant-e2e-shared-a-f9f50d/sub-test-model-f9f50d-kserve-84b54f49c5-ct8md"
Jul 16 01:17:20.798161 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:17:20.798148    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4dda17f-9fe1-4aee-a1b8-003c5be77ba0-home\") pod \"sub-test-model-f9f50d-kserve-84b54f49c5-ct8md\" (UID: \"f4dda17f-9fe1-4aee-a1b8-003c5be77ba0\") " pod="ai-tenant-e2e-shared-a-f9f50d/sub-test-model-f9f50d-kserve-84b54f49c5-ct8md"
Jul 16 01:17:20.899462 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:17:20.899424    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4dda17f-9fe1-4aee-a1b8-003c5be77ba0-model-cache\") pod \"sub-test-model-f9f50d-kserve-84b54f49c5-ct8md\" (UID: \"f4dda17f-9fe1-4aee-a1b8-003c5be77ba0\") " pod="ai-tenant-e2e-shared-a-f9f50d/sub-test-model-f9f50d-kserve-84b54f49c5-ct8md"
Jul 16 01:17:20.899618 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:17:20.899473    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4dda17f-9fe1-4aee-a1b8-003c5be77ba0-tmp-dir\") pod \"sub-test-model-f9f50d-kserve-84b54f49c5-ct8md\" (UID: \"f4dda17f-9fe1-4aee-a1b8-003c5be77ba0\") " pod="ai-tenant-e2e-shared-a-f9f50d/sub-test-model-f9f50d-kserve-84b54f49c5-ct8md"
Jul 16 01:17:20.899618 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:17:20.899505    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcrrw\" (UniqueName: \"kubernetes.io/projected/f4dda17f-9fe1-4aee-a1b8-003c5be77ba0-kube-api-access-vcrrw\") pod \"sub-test-model-f9f50d-kserve-84b54f49c5-ct8md\" (UID: \"f4dda17f-9fe1-4aee-a1b8-003c5be77ba0\") " pod="ai-tenant-e2e-shared-a-f9f50d/sub-test-model-f9f50d-kserve-84b54f49c5-ct8md"
Jul 16 01:17:20.899618 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:17:20.899557    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4dda17f-9fe1-4aee-a1b8-003c5be77ba0-tls-certs\") pod \"sub-test-model-f9f50d-kserve-84b54f49c5-ct8md\" (UID: \"f4dda17f-9fe1-4aee-a1b8-003c5be77ba0\") " pod="ai-tenant-e2e-shared-a-f9f50d/sub-test-model-f9f50d-kserve-84b54f49c5-ct8md"
Jul 16 01:17:20.899618 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:17:20.899596    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4dda17f-9fe1-4aee-a1b8-003c5be77ba0-home\") pod \"sub-test-model-f9f50d-kserve-84b54f49c5-ct8md\" (UID: \"f4dda17f-9fe1-4aee-a1b8-003c5be77ba0\") " pod="ai-tenant-e2e-shared-a-f9f50d/sub-test-model-f9f50d-kserve-84b54f49c5-ct8md"
Jul 16 01:17:20.899858 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:17:20.899627    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4dda17f-9fe1-4aee-a1b8-003c5be77ba0-dshm\") pod \"sub-test-model-f9f50d-kserve-84b54f49c5-ct8md\" (UID: \"f4dda17f-9fe1-4aee-a1b8-003c5be77ba0\") " pod="ai-tenant-e2e-shared-a-f9f50d/sub-test-model-f9f50d-kserve-84b54f49c5-ct8md"
Jul 16 01:17:20.899956 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:17:20.899931    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4dda17f-9fe1-4aee-a1b8-003c5be77ba0-tmp-dir\") pod \"sub-test-model-f9f50d-kserve-84b54f49c5-ct8md\" (UID: \"f4dda17f-9fe1-4aee-a1b8-003c5be77ba0\") " pod="ai-tenant-e2e-shared-a-f9f50d/sub-test-model-f9f50d-kserve-84b54f49c5-ct8md"
Jul 16 01:17:20.900014 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:17:20.899971    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4dda17f-9fe1-4aee-a1b8-003c5be77ba0-model-cache\") pod \"sub-test-model-f9f50d-kserve-84b54f49c5-ct8md\" (UID: \"f4dda17f-9fe1-4aee-a1b8-003c5be77ba0\") " pod="ai-tenant-e2e-shared-a-f9f50d/sub-test-model-f9f50d-kserve-84b54f49c5-ct8md"
Jul 16 01:17:20.900014 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:17:20.900000    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4dda17f-9fe1-4aee-a1b8-003c5be77ba0-home\") pod \"sub-test-model-f9f50d-kserve-84b54f49c5-ct8md\" (UID: \"f4dda17f-9fe1-4aee-a1b8-003c5be77ba0\") " pod="ai-tenant-e2e-shared-a-f9f50d/sub-test-model-f9f50d-kserve-84b54f49c5-ct8md"
Jul 16 01:17:20.901888 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:17:20.901854    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4dda17f-9fe1-4aee-a1b8-003c5be77ba0-dshm\") pod \"sub-test-model-f9f50d-kserve-84b54f49c5-ct8md\" (UID: \"f4dda17f-9fe1-4aee-a1b8-003c5be77ba0\") " pod="ai-tenant-e2e-shared-a-f9f50d/sub-test-model-f9f50d-kserve-84b54f49c5-ct8md"
Jul 16 01:17:20.902061 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:17:20.902045    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4dda17f-9fe1-4aee-a1b8-003c5be77ba0-tls-certs\") pod \"sub-test-model-f9f50d-kserve-84b54f49c5-ct8md\" (UID: \"f4dda17f-9fe1-4aee-a1b8-003c5be77ba0\") " pod="ai-tenant-e2e-shared-a-f9f50d/sub-test-model-f9f50d-kserve-84b54f49c5-ct8md"
Jul 16 01:17:20.908040 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:17:20.907984    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcrrw\" (UniqueName: \"kubernetes.io/projected/f4dda17f-9fe1-4aee-a1b8-003c5be77ba0-kube-api-access-vcrrw\") pod \"sub-test-model-f9f50d-kserve-84b54f49c5-ct8md\" (UID: \"f4dda17f-9fe1-4aee-a1b8-003c5be77ba0\") " pod="ai-tenant-e2e-shared-a-f9f50d/sub-test-model-f9f50d-kserve-84b54f49c5-ct8md"
Jul 16 01:17:20.966087 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:17:20.966043    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-f9f50d/sub-test-model-f9f50d-kserve-84b54f49c5-ct8md"
Jul 16 01:17:21.102414 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:17:21.102391    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-f9f50d/sub-test-model-f9f50d-kserve-84b54f49c5-ct8md"]
Jul 16 01:17:21.105000 ip-10-0-143-65 kubenswrapper[2563]: W0716 01:17:21.104976    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf4dda17f_9fe1_4aee_a1b8_003c5be77ba0.slice/crio-d5473ec5c67d3433366633de2e75a004dd067e0b7e5767909d0e280dd44102e9 WatchSource:0}: Error finding container d5473ec5c67d3433366633de2e75a004dd067e0b7e5767909d0e280dd44102e9: Status 404 returned error can't find the container with id d5473ec5c67d3433366633de2e75a004dd067e0b7e5767909d0e280dd44102e9
Jul 16 01:17:22.068372 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:17:22.068337    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-f9f50d/sub-test-model-f9f50d-kserve-84b54f49c5-ct8md" event={"ID":"f4dda17f-9fe1-4aee-a1b8-003c5be77ba0","Type":"ContainerStarted","Data":"9383548471c299fb08fec399ac6b41e64d2c13fe6c600a6e93c21dc76d8d0baf"}
Jul 16 01:17:22.068718 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:17:22.068379    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-f9f50d/sub-test-model-f9f50d-kserve-84b54f49c5-ct8md" event={"ID":"f4dda17f-9fe1-4aee-a1b8-003c5be77ba0","Type":"ContainerStarted","Data":"d5473ec5c67d3433366633de2e75a004dd067e0b7e5767909d0e280dd44102e9"}
Jul 16 01:17:22.085262 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:17:22.085214    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-f9f50d/sub-test-model-f9f50d-kserve-84b54f49c5-ct8md" podStartSLOduration=2.085199368 podStartE2EDuration="2.085199368s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 01:17:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 01:17:22.084944209 +0000 UTC m=+2214.520205949" watchObservedRunningTime="2026-07-16 01:17:22.085199368 +0000 UTC m=+2214.520461104"
Jul 16 01:17:24.858039 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:17:24.858008    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl7fx"]
Jul 16 01:17:30.966278 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:17:30.966241    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-f9f50d/sub-test-model-f9f50d-kserve-84b54f49c5-ct8md"
Jul 16 01:17:30.966760 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:17:30.966301    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-f9f50d/sub-test-model-f9f50d-kserve-84b54f49c5-ct8md"
Jul 16 01:17:30.979310 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:17:30.979276    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-f9f50d/sub-test-model-f9f50d-kserve-84b54f49c5-ct8md"
Jul 16 01:17:31.120331 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:17:31.120304    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-f9f50d/sub-test-model-f9f50d-kserve-84b54f49c5-ct8md"
Jul 16 01:17:55.659750 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:17:55.659715    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-628c71/sub-test-model-628c71-kserve-67988bfbf-qns56"]
Jul 16 01:17:55.663095 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:17:55.663079    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-628c71/sub-test-model-628c71-kserve-67988bfbf-qns56"
Jul 16 01:17:55.666061 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:17:55.666041    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-628c71\"/\"openshift-service-ca.crt\""
Jul 16 01:17:55.666061 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:17:55.666043    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-628c71\"/\"sub-test-model-628c71-kserve-self-signed-certs\""
Jul 16 01:17:55.666229 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:17:55.666047    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-628c71\"/\"default-dockercfg-f9hx6\""
Jul 16 01:17:55.666229 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:17:55.666087    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-628c71\"/\"kube-root-ca.crt\""
Jul 16 01:17:55.673794 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:17:55.673773    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-628c71/sub-test-model-628c71-kserve-67988bfbf-qns56"]
Jul 16 01:17:55.835625 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:17:55.835587    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c641b326-4164-438e-83fd-5a0b2b21879d-dshm\") pod \"sub-test-model-628c71-kserve-67988bfbf-qns56\" (UID: \"c641b326-4164-438e-83fd-5a0b2b21879d\") " pod="ai-tenant-e2e-shared-b-628c71/sub-test-model-628c71-kserve-67988bfbf-qns56"
Jul 16 01:17:55.835834 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:17:55.835681    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c641b326-4164-438e-83fd-5a0b2b21879d-tls-certs\") pod \"sub-test-model-628c71-kserve-67988bfbf-qns56\" (UID: \"c641b326-4164-438e-83fd-5a0b2b21879d\") " pod="ai-tenant-e2e-shared-b-628c71/sub-test-model-628c71-kserve-67988bfbf-qns56"
Jul 16 01:17:55.835834 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:17:55.835708    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgd9f\" (UniqueName: \"kubernetes.io/projected/c641b326-4164-438e-83fd-5a0b2b21879d-kube-api-access-xgd9f\") pod \"sub-test-model-628c71-kserve-67988bfbf-qns56\" (UID: \"c641b326-4164-438e-83fd-5a0b2b21879d\") " pod="ai-tenant-e2e-shared-b-628c71/sub-test-model-628c71-kserve-67988bfbf-qns56"
Jul 16 01:17:55.835834 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:17:55.835800    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c641b326-4164-438e-83fd-5a0b2b21879d-model-cache\") pod \"sub-test-model-628c71-kserve-67988bfbf-qns56\" (UID: \"c641b326-4164-438e-83fd-5a0b2b21879d\") " pod="ai-tenant-e2e-shared-b-628c71/sub-test-model-628c71-kserve-67988bfbf-qns56"
Jul 16 01:17:55.836018 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:17:55.835837    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c641b326-4164-438e-83fd-5a0b2b21879d-home\") pod \"sub-test-model-628c71-kserve-67988bfbf-qns56\" (UID: \"c641b326-4164-438e-83fd-5a0b2b21879d\") " pod="ai-tenant-e2e-shared-b-628c71/sub-test-model-628c71-kserve-67988bfbf-qns56"
Jul 16 01:17:55.836018 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:17:55.835935    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c641b326-4164-438e-83fd-5a0b2b21879d-tmp-dir\") pod \"sub-test-model-628c71-kserve-67988bfbf-qns56\" (UID: \"c641b326-4164-438e-83fd-5a0b2b21879d\") " pod="ai-tenant-e2e-shared-b-628c71/sub-test-model-628c71-kserve-67988bfbf-qns56"
Jul 16 01:17:55.936450 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:17:55.936357    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c641b326-4164-438e-83fd-5a0b2b21879d-tmp-dir\") pod \"sub-test-model-628c71-kserve-67988bfbf-qns56\" (UID: \"c641b326-4164-438e-83fd-5a0b2b21879d\") " pod="ai-tenant-e2e-shared-b-628c71/sub-test-model-628c71-kserve-67988bfbf-qns56"
Jul 16 01:17:55.936450 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:17:55.936399    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c641b326-4164-438e-83fd-5a0b2b21879d-dshm\") pod \"sub-test-model-628c71-kserve-67988bfbf-qns56\" (UID: \"c641b326-4164-438e-83fd-5a0b2b21879d\") " pod="ai-tenant-e2e-shared-b-628c71/sub-test-model-628c71-kserve-67988bfbf-qns56"
Jul 16 01:17:55.936450 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:17:55.936448    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c641b326-4164-438e-83fd-5a0b2b21879d-tls-certs\") pod \"sub-test-model-628c71-kserve-67988bfbf-qns56\" (UID: \"c641b326-4164-438e-83fd-5a0b2b21879d\") " pod="ai-tenant-e2e-shared-b-628c71/sub-test-model-628c71-kserve-67988bfbf-qns56"
Jul 16 01:17:55.936723 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:17:55.936466    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgd9f\" (UniqueName: \"kubernetes.io/projected/c641b326-4164-438e-83fd-5a0b2b21879d-kube-api-access-xgd9f\") pod \"sub-test-model-628c71-kserve-67988bfbf-qns56\" (UID: \"c641b326-4164-438e-83fd-5a0b2b21879d\") " pod="ai-tenant-e2e-shared-b-628c71/sub-test-model-628c71-kserve-67988bfbf-qns56"
Jul 16 01:17:55.936723 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:17:55.936510    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c641b326-4164-438e-83fd-5a0b2b21879d-model-cache\") pod \"sub-test-model-628c71-kserve-67988bfbf-qns56\" (UID: \"c641b326-4164-438e-83fd-5a0b2b21879d\") " pod="ai-tenant-e2e-shared-b-628c71/sub-test-model-628c71-kserve-67988bfbf-qns56"
Jul 16 01:17:55.936723 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:17:55.936542    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c641b326-4164-438e-83fd-5a0b2b21879d-home\") pod \"sub-test-model-628c71-kserve-67988bfbf-qns56\" (UID: \"c641b326-4164-438e-83fd-5a0b2b21879d\") " pod="ai-tenant-e2e-shared-b-628c71/sub-test-model-628c71-kserve-67988bfbf-qns56"
Jul 16 01:17:55.936911 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:17:55.936830    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c641b326-4164-438e-83fd-5a0b2b21879d-tmp-dir\") pod \"sub-test-model-628c71-kserve-67988bfbf-qns56\" (UID: \"c641b326-4164-438e-83fd-5a0b2b21879d\") " pod="ai-tenant-e2e-shared-b-628c71/sub-test-model-628c71-kserve-67988bfbf-qns56"
Jul 16 01:17:55.936958 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:17:55.936929    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c641b326-4164-438e-83fd-5a0b2b21879d-model-cache\") pod \"sub-test-model-628c71-kserve-67988bfbf-qns56\" (UID: \"c641b326-4164-438e-83fd-5a0b2b21879d\") " pod="ai-tenant-e2e-shared-b-628c71/sub-test-model-628c71-kserve-67988bfbf-qns56"
Jul 16 01:17:55.936993 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:17:55.936976    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c641b326-4164-438e-83fd-5a0b2b21879d-home\") pod \"sub-test-model-628c71-kserve-67988bfbf-qns56\" (UID: \"c641b326-4164-438e-83fd-5a0b2b21879d\") " pod="ai-tenant-e2e-shared-b-628c71/sub-test-model-628c71-kserve-67988bfbf-qns56"
Jul 16 01:17:55.938813 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:17:55.938793    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c641b326-4164-438e-83fd-5a0b2b21879d-dshm\") pod \"sub-test-model-628c71-kserve-67988bfbf-qns56\" (UID: \"c641b326-4164-438e-83fd-5a0b2b21879d\") " pod="ai-tenant-e2e-shared-b-628c71/sub-test-model-628c71-kserve-67988bfbf-qns56"
Jul 16 01:17:55.938989 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:17:55.938973    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c641b326-4164-438e-83fd-5a0b2b21879d-tls-certs\") pod \"sub-test-model-628c71-kserve-67988bfbf-qns56\" (UID: \"c641b326-4164-438e-83fd-5a0b2b21879d\") " pod="ai-tenant-e2e-shared-b-628c71/sub-test-model-628c71-kserve-67988bfbf-qns56"
Jul 16 01:17:55.945203 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:17:55.945184    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgd9f\" (UniqueName: \"kubernetes.io/projected/c641b326-4164-438e-83fd-5a0b2b21879d-kube-api-access-xgd9f\") pod \"sub-test-model-628c71-kserve-67988bfbf-qns56\" (UID: \"c641b326-4164-438e-83fd-5a0b2b21879d\") " pod="ai-tenant-e2e-shared-b-628c71/sub-test-model-628c71-kserve-67988bfbf-qns56"
Jul 16 01:17:55.972181 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:17:55.972158    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-628c71/sub-test-model-628c71-kserve-67988bfbf-qns56"
Jul 16 01:17:56.098372 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:17:56.098347    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-628c71/sub-test-model-628c71-kserve-67988bfbf-qns56"]
Jul 16 01:17:56.100531 ip-10-0-143-65 kubenswrapper[2563]: W0716 01:17:56.100506    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc641b326_4164_438e_83fd_5a0b2b21879d.slice/crio-ee8d7f4a35b0b06490b36d42cb407d5bff9814e89d5153787ded21ad2a0f4c40 WatchSource:0}: Error finding container ee8d7f4a35b0b06490b36d42cb407d5bff9814e89d5153787ded21ad2a0f4c40: Status 404 returned error can't find the container with id ee8d7f4a35b0b06490b36d42cb407d5bff9814e89d5153787ded21ad2a0f4c40
Jul 16 01:17:56.233005 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:17:56.232912    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-628c71/sub-test-model-628c71-kserve-67988bfbf-qns56" event={"ID":"c641b326-4164-438e-83fd-5a0b2b21879d","Type":"ContainerStarted","Data":"c0bfc34aa3f12dc1efc2931e33b482ceba258e733b25c3a80bb2c1082a78e0d7"}
Jul 16 01:17:56.233005 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:17:56.232963    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-628c71/sub-test-model-628c71-kserve-67988bfbf-qns56" event={"ID":"c641b326-4164-438e-83fd-5a0b2b21879d","Type":"ContainerStarted","Data":"ee8d7f4a35b0b06490b36d42cb407d5bff9814e89d5153787ded21ad2a0f4c40"}
Jul 16 01:17:56.253614 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:17:56.253555    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-628c71/sub-test-model-628c71-kserve-67988bfbf-qns56" podStartSLOduration=1.253539719 podStartE2EDuration="1.253539719s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 01:17:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 01:17:56.251583408 +0000 UTC m=+2248.686845149" watchObservedRunningTime="2026-07-16 01:17:56.253539719 +0000 UTC m=+2248.688801458"
Jul 16 01:18:01.156762 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:18:01.156716    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl7fx"]
Jul 16 01:18:05.972554 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:18:05.972502    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-628c71/sub-test-model-628c71-kserve-67988bfbf-qns56"
Jul 16 01:18:05.972554 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:18:05.972573    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-628c71/sub-test-model-628c71-kserve-67988bfbf-qns56"
Jul 16 01:18:05.985184 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:18:05.985152    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-628c71/sub-test-model-628c71-kserve-67988bfbf-qns56"
Jul 16 01:18:06.293369 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:18:06.293290    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-628c71/sub-test-model-628c71-kserve-67988bfbf-qns56"
Jul 16 01:18:17.841899 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:18:17.841849    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl7fx"]
Jul 16 01:18:35.798824 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:18:35.798785    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-f9f50d/sub-test-model-f9f50d-kserve-84b54f49c5-ct8md"]
Jul 16 01:18:36.732393 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:18:36.732359    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-628c71/sub-test-model-628c71-kserve-67988bfbf-qns56"]
Jul 16 01:18:40.745050 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:18:40.745016    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl7fx"]
Jul 16 01:18:41.755948 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:18:41.755910    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-f9f50d/rate-test-model-f9f50d-kserve-79b8d64ccf-rch5r"]
Jul 16 01:18:41.759465 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:18:41.759446    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-f9f50d/rate-test-model-f9f50d-kserve-79b8d64ccf-rch5r"
Jul 16 01:18:41.761516 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:18:41.761495    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-f9f50d\"/\"rate-test-model-f9f50d-kserve-self-signed-certs\""
Jul 16 01:18:41.768093 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:18:41.768065    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-f9f50d/rate-test-model-f9f50d-kserve-79b8d64ccf-rch5r"]
Jul 16 01:18:41.781723 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:18:41.781695    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/189327f6-723c-44fe-b770-9c3b049dab41-model-cache\") pod \"rate-test-model-f9f50d-kserve-79b8d64ccf-rch5r\" (UID: \"189327f6-723c-44fe-b770-9c3b049dab41\") " pod="ai-tenant-e2e-shared-a-f9f50d/rate-test-model-f9f50d-kserve-79b8d64ccf-rch5r"
Jul 16 01:18:41.781844 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:18:41.781734    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/189327f6-723c-44fe-b770-9c3b049dab41-dshm\") pod \"rate-test-model-f9f50d-kserve-79b8d64ccf-rch5r\" (UID: \"189327f6-723c-44fe-b770-9c3b049dab41\") " pod="ai-tenant-e2e-shared-a-f9f50d/rate-test-model-f9f50d-kserve-79b8d64ccf-rch5r"
Jul 16 01:18:41.781844 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:18:41.781802    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/189327f6-723c-44fe-b770-9c3b049dab41-tmp-dir\") pod \"rate-test-model-f9f50d-kserve-79b8d64ccf-rch5r\" (UID: \"189327f6-723c-44fe-b770-9c3b049dab41\") " pod="ai-tenant-e2e-shared-a-f9f50d/rate-test-model-f9f50d-kserve-79b8d64ccf-rch5r"
Jul 16 01:18:41.781958 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:18:41.781886    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/189327f6-723c-44fe-b770-9c3b049dab41-tls-certs\") pod \"rate-test-model-f9f50d-kserve-79b8d64ccf-rch5r\" (UID: \"189327f6-723c-44fe-b770-9c3b049dab41\") " pod="ai-tenant-e2e-shared-a-f9f50d/rate-test-model-f9f50d-kserve-79b8d64ccf-rch5r"
Jul 16 01:18:41.781958 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:18:41.781903    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4hqz\" (UniqueName: \"kubernetes.io/projected/189327f6-723c-44fe-b770-9c3b049dab41-kube-api-access-f4hqz\") pod \"rate-test-model-f9f50d-kserve-79b8d64ccf-rch5r\" (UID: \"189327f6-723c-44fe-b770-9c3b049dab41\") " pod="ai-tenant-e2e-shared-a-f9f50d/rate-test-model-f9f50d-kserve-79b8d64ccf-rch5r"
Jul 16 01:18:41.781958 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:18:41.781930    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/189327f6-723c-44fe-b770-9c3b049dab41-home\") pod \"rate-test-model-f9f50d-kserve-79b8d64ccf-rch5r\" (UID: \"189327f6-723c-44fe-b770-9c3b049dab41\") " pod="ai-tenant-e2e-shared-a-f9f50d/rate-test-model-f9f50d-kserve-79b8d64ccf-rch5r"
Jul 16 01:18:41.883090 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:18:41.883051    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/189327f6-723c-44fe-b770-9c3b049dab41-tls-certs\") pod \"rate-test-model-f9f50d-kserve-79b8d64ccf-rch5r\" (UID: \"189327f6-723c-44fe-b770-9c3b049dab41\") " pod="ai-tenant-e2e-shared-a-f9f50d/rate-test-model-f9f50d-kserve-79b8d64ccf-rch5r"
Jul 16 01:18:41.883090 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:18:41.883088    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4hqz\" (UniqueName: \"kubernetes.io/projected/189327f6-723c-44fe-b770-9c3b049dab41-kube-api-access-f4hqz\") pod \"rate-test-model-f9f50d-kserve-79b8d64ccf-rch5r\" (UID: \"189327f6-723c-44fe-b770-9c3b049dab41\") " pod="ai-tenant-e2e-shared-a-f9f50d/rate-test-model-f9f50d-kserve-79b8d64ccf-rch5r"
Jul 16 01:18:41.883358 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:18:41.883112    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/189327f6-723c-44fe-b770-9c3b049dab41-home\") pod \"rate-test-model-f9f50d-kserve-79b8d64ccf-rch5r\" (UID: \"189327f6-723c-44fe-b770-9c3b049dab41\") " pod="ai-tenant-e2e-shared-a-f9f50d/rate-test-model-f9f50d-kserve-79b8d64ccf-rch5r"
Jul 16 01:18:41.883358 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:18:41.883169    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/189327f6-723c-44fe-b770-9c3b049dab41-model-cache\") pod \"rate-test-model-f9f50d-kserve-79b8d64ccf-rch5r\" (UID: \"189327f6-723c-44fe-b770-9c3b049dab41\") " pod="ai-tenant-e2e-shared-a-f9f50d/rate-test-model-f9f50d-kserve-79b8d64ccf-rch5r"
Jul 16 01:18:41.883358 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:18:41.883190    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/189327f6-723c-44fe-b770-9c3b049dab41-dshm\") pod \"rate-test-model-f9f50d-kserve-79b8d64ccf-rch5r\" (UID: \"189327f6-723c-44fe-b770-9c3b049dab41\") " pod="ai-tenant-e2e-shared-a-f9f50d/rate-test-model-f9f50d-kserve-79b8d64ccf-rch5r"
Jul 16 01:18:41.883358 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:18:41.883326    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/189327f6-723c-44fe-b770-9c3b049dab41-tmp-dir\") pod \"rate-test-model-f9f50d-kserve-79b8d64ccf-rch5r\" (UID: \"189327f6-723c-44fe-b770-9c3b049dab41\") " pod="ai-tenant-e2e-shared-a-f9f50d/rate-test-model-f9f50d-kserve-79b8d64ccf-rch5r"
Jul 16 01:18:41.883622 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:18:41.883603    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/189327f6-723c-44fe-b770-9c3b049dab41-home\") pod \"rate-test-model-f9f50d-kserve-79b8d64ccf-rch5r\" (UID: \"189327f6-723c-44fe-b770-9c3b049dab41\") " pod="ai-tenant-e2e-shared-a-f9f50d/rate-test-model-f9f50d-kserve-79b8d64ccf-rch5r"
Jul 16 01:18:41.883667 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:18:41.883630    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/189327f6-723c-44fe-b770-9c3b049dab41-model-cache\") pod \"rate-test-model-f9f50d-kserve-79b8d64ccf-rch5r\" (UID: \"189327f6-723c-44fe-b770-9c3b049dab41\") " pod="ai-tenant-e2e-shared-a-f9f50d/rate-test-model-f9f50d-kserve-79b8d64ccf-rch5r"
Jul 16 01:18:41.883705 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:18:41.883673    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/189327f6-723c-44fe-b770-9c3b049dab41-tmp-dir\") pod \"rate-test-model-f9f50d-kserve-79b8d64ccf-rch5r\" (UID: \"189327f6-723c-44fe-b770-9c3b049dab41\") " pod="ai-tenant-e2e-shared-a-f9f50d/rate-test-model-f9f50d-kserve-79b8d64ccf-rch5r"
Jul 16 01:18:41.885456 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:18:41.885437    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/189327f6-723c-44fe-b770-9c3b049dab41-dshm\") pod \"rate-test-model-f9f50d-kserve-79b8d64ccf-rch5r\" (UID: \"189327f6-723c-44fe-b770-9c3b049dab41\") " pod="ai-tenant-e2e-shared-a-f9f50d/rate-test-model-f9f50d-kserve-79b8d64ccf-rch5r"
Jul 16 01:18:41.885630 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:18:41.885613    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/189327f6-723c-44fe-b770-9c3b049dab41-tls-certs\") pod \"rate-test-model-f9f50d-kserve-79b8d64ccf-rch5r\" (UID: \"189327f6-723c-44fe-b770-9c3b049dab41\") " pod="ai-tenant-e2e-shared-a-f9f50d/rate-test-model-f9f50d-kserve-79b8d64ccf-rch5r"
Jul 16 01:18:41.893712 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:18:41.893686    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4hqz\" (UniqueName: \"kubernetes.io/projected/189327f6-723c-44fe-b770-9c3b049dab41-kube-api-access-f4hqz\") pod \"rate-test-model-f9f50d-kserve-79b8d64ccf-rch5r\" (UID: \"189327f6-723c-44fe-b770-9c3b049dab41\") " pod="ai-tenant-e2e-shared-a-f9f50d/rate-test-model-f9f50d-kserve-79b8d64ccf-rch5r"
Jul 16 01:18:42.068718 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:18:42.068623    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-f9f50d/rate-test-model-f9f50d-kserve-79b8d64ccf-rch5r"
Jul 16 01:18:42.200769 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:18:42.200740    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-f9f50d/rate-test-model-f9f50d-kserve-79b8d64ccf-rch5r"]
Jul 16 01:18:42.203377 ip-10-0-143-65 kubenswrapper[2563]: W0716 01:18:42.203346    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod189327f6_723c_44fe_b770_9c3b049dab41.slice/crio-294d6b64bc2afe96019ea4ed462fcd5712539793045b3b1c3145ab12a1ebf574 WatchSource:0}: Error finding container 294d6b64bc2afe96019ea4ed462fcd5712539793045b3b1c3145ab12a1ebf574: Status 404 returned error can't find the container with id 294d6b64bc2afe96019ea4ed462fcd5712539793045b3b1c3145ab12a1ebf574
Jul 16 01:18:42.447326 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:18:42.447292    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-f9f50d/rate-test-model-f9f50d-kserve-79b8d64ccf-rch5r" event={"ID":"189327f6-723c-44fe-b770-9c3b049dab41","Type":"ContainerStarted","Data":"ad639821c4441e4038d0a820a37eca1ef03e7a24360cfc2b8dce74f9b68d5db0"}
Jul 16 01:18:42.447326 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:18:42.447332    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-f9f50d/rate-test-model-f9f50d-kserve-79b8d64ccf-rch5r" event={"ID":"189327f6-723c-44fe-b770-9c3b049dab41","Type":"ContainerStarted","Data":"294d6b64bc2afe96019ea4ed462fcd5712539793045b3b1c3145ab12a1ebf574"}
Jul 16 01:18:42.465145 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:18:42.465097    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-f9f50d/rate-test-model-f9f50d-kserve-79b8d64ccf-rch5r" podStartSLOduration=1.465080767 podStartE2EDuration="1.465080767s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 01:18:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 01:18:42.463026906 +0000 UTC m=+2294.898288644" watchObservedRunningTime="2026-07-16 01:18:42.465080767 +0000 UTC m=+2294.900342506"
Jul 16 01:18:50.819102 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:18:50.819044    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-f9f50d/sub-test-model-f9f50d-kserve-84b54f49c5-ct8md" podUID="f4dda17f-9fe1-4aee-a1b8-003c5be77ba0" containerName="main" containerID="cri-o://9383548471c299fb08fec399ac6b41e64d2c13fe6c600a6e93c21dc76d8d0baf" gracePeriod=45
Jul 16 01:18:51.073466 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:18:51.073438    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-f9f50d/sub-test-model-f9f50d-kserve-84b54f49c5-ct8md"
Jul 16 01:18:51.161386 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:18:51.161348    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4dda17f-9fe1-4aee-a1b8-003c5be77ba0-dshm\") pod \"f4dda17f-9fe1-4aee-a1b8-003c5be77ba0\" (UID: \"f4dda17f-9fe1-4aee-a1b8-003c5be77ba0\") "
Jul 16 01:18:51.161594 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:18:51.161400    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vcrrw\" (UniqueName: \"kubernetes.io/projected/f4dda17f-9fe1-4aee-a1b8-003c5be77ba0-kube-api-access-vcrrw\") pod \"f4dda17f-9fe1-4aee-a1b8-003c5be77ba0\" (UID: \"f4dda17f-9fe1-4aee-a1b8-003c5be77ba0\") "
Jul 16 01:18:51.161594 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:18:51.161442    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4dda17f-9fe1-4aee-a1b8-003c5be77ba0-model-cache\") pod \"f4dda17f-9fe1-4aee-a1b8-003c5be77ba0\" (UID: \"f4dda17f-9fe1-4aee-a1b8-003c5be77ba0\") "
Jul 16 01:18:51.161594 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:18:51.161496    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4dda17f-9fe1-4aee-a1b8-003c5be77ba0-tmp-dir\") pod \"f4dda17f-9fe1-4aee-a1b8-003c5be77ba0\" (UID: \"f4dda17f-9fe1-4aee-a1b8-003c5be77ba0\") "
Jul 16 01:18:51.161594 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:18:51.161519    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4dda17f-9fe1-4aee-a1b8-003c5be77ba0-home\") pod \"f4dda17f-9fe1-4aee-a1b8-003c5be77ba0\" (UID: \"f4dda17f-9fe1-4aee-a1b8-003c5be77ba0\") "
Jul 16 01:18:51.161594 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:18:51.161536    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4dda17f-9fe1-4aee-a1b8-003c5be77ba0-tls-certs\") pod \"f4dda17f-9fe1-4aee-a1b8-003c5be77ba0\" (UID: \"f4dda17f-9fe1-4aee-a1b8-003c5be77ba0\") "
Jul 16 01:18:51.161893 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:18:51.161731    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4dda17f-9fe1-4aee-a1b8-003c5be77ba0-model-cache" (OuterVolumeSpecName: "model-cache") pod "f4dda17f-9fe1-4aee-a1b8-003c5be77ba0" (UID: "f4dda17f-9fe1-4aee-a1b8-003c5be77ba0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 01:18:51.161893 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:18:51.161746    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4dda17f-9fe1-4aee-a1b8-003c5be77ba0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f4dda17f-9fe1-4aee-a1b8-003c5be77ba0" (UID: "f4dda17f-9fe1-4aee-a1b8-003c5be77ba0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 01:18:51.161975 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:18:51.161913    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4dda17f-9fe1-4aee-a1b8-003c5be77ba0-home" (OuterVolumeSpecName: "home") pod "f4dda17f-9fe1-4aee-a1b8-003c5be77ba0" (UID: "f4dda17f-9fe1-4aee-a1b8-003c5be77ba0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 01:18:51.163614 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:18:51.163584    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4dda17f-9fe1-4aee-a1b8-003c5be77ba0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f4dda17f-9fe1-4aee-a1b8-003c5be77ba0" (UID: "f4dda17f-9fe1-4aee-a1b8-003c5be77ba0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 01:18:51.163725 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:18:51.163642    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4dda17f-9fe1-4aee-a1b8-003c5be77ba0-kube-api-access-vcrrw" (OuterVolumeSpecName: "kube-api-access-vcrrw") pod "f4dda17f-9fe1-4aee-a1b8-003c5be77ba0" (UID: "f4dda17f-9fe1-4aee-a1b8-003c5be77ba0"). InnerVolumeSpecName "kube-api-access-vcrrw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 01:18:51.163725 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:18:51.163661    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4dda17f-9fe1-4aee-a1b8-003c5be77ba0-dshm" (OuterVolumeSpecName: "dshm") pod "f4dda17f-9fe1-4aee-a1b8-003c5be77ba0" (UID: "f4dda17f-9fe1-4aee-a1b8-003c5be77ba0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 01:18:51.263194 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:18:51.263158    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vcrrw\" (UniqueName: \"kubernetes.io/projected/f4dda17f-9fe1-4aee-a1b8-003c5be77ba0-kube-api-access-vcrrw\") on node \"ip-10-0-143-65.ec2.internal\" DevicePath \"\""
Jul 16 01:18:51.263194 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:18:51.263189    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4dda17f-9fe1-4aee-a1b8-003c5be77ba0-model-cache\") on node \"ip-10-0-143-65.ec2.internal\" DevicePath \"\""
Jul 16 01:18:51.263194 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:18:51.263199    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4dda17f-9fe1-4aee-a1b8-003c5be77ba0-tmp-dir\") on node \"ip-10-0-143-65.ec2.internal\" DevicePath \"\""
Jul 16 01:18:51.263194 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:18:51.263206    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4dda17f-9fe1-4aee-a1b8-003c5be77ba0-home\") on node \"ip-10-0-143-65.ec2.internal\" DevicePath \"\""
Jul 16 01:18:51.263461 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:18:51.263214    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4dda17f-9fe1-4aee-a1b8-003c5be77ba0-tls-certs\") on node \"ip-10-0-143-65.ec2.internal\" DevicePath \"\""
Jul 16 01:18:51.263461 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:18:51.263224    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4dda17f-9fe1-4aee-a1b8-003c5be77ba0-dshm\") on node \"ip-10-0-143-65.ec2.internal\" DevicePath \"\""
Jul 16 01:18:51.497978 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:18:51.497888    2563 generic.go:356] "Generic (PLEG): container finished" podID="f4dda17f-9fe1-4aee-a1b8-003c5be77ba0" containerID="9383548471c299fb08fec399ac6b41e64d2c13fe6c600a6e93c21dc76d8d0baf" exitCode=0
Jul 16 01:18:51.497978 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:18:51.497947    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-f9f50d/sub-test-model-f9f50d-kserve-84b54f49c5-ct8md"
Jul 16 01:18:51.498189 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:18:51.497971    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-f9f50d/sub-test-model-f9f50d-kserve-84b54f49c5-ct8md" event={"ID":"f4dda17f-9fe1-4aee-a1b8-003c5be77ba0","Type":"ContainerDied","Data":"9383548471c299fb08fec399ac6b41e64d2c13fe6c600a6e93c21dc76d8d0baf"}
Jul 16 01:18:51.498189 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:18:51.498017    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-f9f50d/sub-test-model-f9f50d-kserve-84b54f49c5-ct8md" event={"ID":"f4dda17f-9fe1-4aee-a1b8-003c5be77ba0","Type":"ContainerDied","Data":"d5473ec5c67d3433366633de2e75a004dd067e0b7e5767909d0e280dd44102e9"}
Jul 16 01:18:51.498189 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:18:51.498037    2563 scope.go:117] "RemoveContainer" containerID="9383548471c299fb08fec399ac6b41e64d2c13fe6c600a6e93c21dc76d8d0baf"
Jul 16 01:18:51.506944 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:18:51.506921    2563 scope.go:117] "RemoveContainer" containerID="9383548471c299fb08fec399ac6b41e64d2c13fe6c600a6e93c21dc76d8d0baf"
Jul 16 01:18:51.507231 ip-10-0-143-65 kubenswrapper[2563]: E0716 01:18:51.507209    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9383548471c299fb08fec399ac6b41e64d2c13fe6c600a6e93c21dc76d8d0baf\": container with ID starting with 9383548471c299fb08fec399ac6b41e64d2c13fe6c600a6e93c21dc76d8d0baf not found: ID does not exist" containerID="9383548471c299fb08fec399ac6b41e64d2c13fe6c600a6e93c21dc76d8d0baf"
Jul 16 01:18:51.507297 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:18:51.507239    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9383548471c299fb08fec399ac6b41e64d2c13fe6c600a6e93c21dc76d8d0baf"} err="failed to get container status \"9383548471c299fb08fec399ac6b41e64d2c13fe6c600a6e93c21dc76d8d0baf\": rpc error: code = NotFound desc = could not find container \"9383548471c299fb08fec399ac6b41e64d2c13fe6c600a6e93c21dc76d8d0baf\": container with ID starting with 9383548471c299fb08fec399ac6b41e64d2c13fe6c600a6e93c21dc76d8d0baf not found: ID does not exist"
Jul 16 01:18:51.522658 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:18:51.522634    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-f9f50d/sub-test-model-f9f50d-kserve-84b54f49c5-ct8md"]
Jul 16 01:18:51.528175 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:18:51.528143    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-f9f50d/sub-test-model-f9f50d-kserve-84b54f49c5-ct8md"]
Jul 16 01:18:51.753174 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:18:51.753055    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-628c71/sub-test-model-628c71-kserve-67988bfbf-qns56" podUID="c641b326-4164-438e-83fd-5a0b2b21879d" containerName="main" containerID="cri-o://c0bfc34aa3f12dc1efc2931e33b482ceba258e733b25c3a80bb2c1082a78e0d7" gracePeriod=45
Jul 16 01:18:52.002728 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:18:52.002693    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-628c71/sub-test-model-628c71-kserve-67988bfbf-qns56"
Jul 16 01:18:52.006816 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:18:52.006746    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f4dda17f-9fe1-4aee-a1b8-003c5be77ba0" path="/var/lib/kubelet/pods/f4dda17f-9fe1-4aee-a1b8-003c5be77ba0/volumes"
Jul 16 01:18:52.069719 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:18:52.069686    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-f9f50d/rate-test-model-f9f50d-kserve-79b8d64ccf-rch5r"
Jul 16 01:18:52.070007 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:18:52.069984    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-f9f50d/rate-test-model-f9f50d-kserve-79b8d64ccf-rch5r"
Jul 16 01:18:52.071330 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:18:52.071305    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c641b326-4164-438e-83fd-5a0b2b21879d-home\") pod \"c641b326-4164-438e-83fd-5a0b2b21879d\" (UID: \"c641b326-4164-438e-83fd-5a0b2b21879d\") "
Jul 16 01:18:52.071440 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:18:52.071373    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c641b326-4164-438e-83fd-5a0b2b21879d-tmp-dir\") pod \"c641b326-4164-438e-83fd-5a0b2b21879d\" (UID: \"c641b326-4164-438e-83fd-5a0b2b21879d\") "
Jul 16 01:18:52.071440 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:18:52.071412    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c641b326-4164-438e-83fd-5a0b2b21879d-model-cache\") pod \"c641b326-4164-438e-83fd-5a0b2b21879d\" (UID: \"c641b326-4164-438e-83fd-5a0b2b21879d\") "
Jul 16 01:18:52.071551 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:18:52.071457    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c641b326-4164-438e-83fd-5a0b2b21879d-tls-certs\") pod \"c641b326-4164-438e-83fd-5a0b2b21879d\" (UID: \"c641b326-4164-438e-83fd-5a0b2b21879d\") "
Jul 16 01:18:52.071551 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:18:52.071514    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c641b326-4164-438e-83fd-5a0b2b21879d-dshm\") pod \"c641b326-4164-438e-83fd-5a0b2b21879d\" (UID: \"c641b326-4164-438e-83fd-5a0b2b21879d\") "
Jul 16 01:18:52.071654 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:18:52.071566    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xgd9f\" (UniqueName: \"kubernetes.io/projected/c641b326-4164-438e-83fd-5a0b2b21879d-kube-api-access-xgd9f\") pod \"c641b326-4164-438e-83fd-5a0b2b21879d\" (UID: \"c641b326-4164-438e-83fd-5a0b2b21879d\") "
Jul 16 01:18:52.071654 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:18:52.071615    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c641b326-4164-438e-83fd-5a0b2b21879d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c641b326-4164-438e-83fd-5a0b2b21879d" (UID: "c641b326-4164-438e-83fd-5a0b2b21879d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 01:18:52.071654 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:18:52.071641    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c641b326-4164-438e-83fd-5a0b2b21879d-home" (OuterVolumeSpecName: "home") pod "c641b326-4164-438e-83fd-5a0b2b21879d" (UID: "c641b326-4164-438e-83fd-5a0b2b21879d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 01:18:52.071819 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:18:52.071769    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c641b326-4164-438e-83fd-5a0b2b21879d-model-cache" (OuterVolumeSpecName: "model-cache") pod "c641b326-4164-438e-83fd-5a0b2b21879d" (UID: "c641b326-4164-438e-83fd-5a0b2b21879d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 01:18:52.071941 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:18:52.071923    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c641b326-4164-438e-83fd-5a0b2b21879d-home\") on node \"ip-10-0-143-65.ec2.internal\" DevicePath \"\""
Jul 16 01:18:52.072025 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:18:52.071970    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c641b326-4164-438e-83fd-5a0b2b21879d-tmp-dir\") on node \"ip-10-0-143-65.ec2.internal\" DevicePath \"\""
Jul 16 01:18:52.072025 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:18:52.071984    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c641b326-4164-438e-83fd-5a0b2b21879d-model-cache\") on node \"ip-10-0-143-65.ec2.internal\" DevicePath \"\""
Jul 16 01:18:52.073798 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:18:52.073774    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c641b326-4164-438e-83fd-5a0b2b21879d-dshm" (OuterVolumeSpecName: "dshm") pod "c641b326-4164-438e-83fd-5a0b2b21879d" (UID: "c641b326-4164-438e-83fd-5a0b2b21879d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 01:18:52.074044 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:18:52.074013    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c641b326-4164-438e-83fd-5a0b2b21879d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c641b326-4164-438e-83fd-5a0b2b21879d" (UID: "c641b326-4164-438e-83fd-5a0b2b21879d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 01:18:52.074152 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:18:52.074105    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c641b326-4164-438e-83fd-5a0b2b21879d-kube-api-access-xgd9f" (OuterVolumeSpecName: "kube-api-access-xgd9f") pod "c641b326-4164-438e-83fd-5a0b2b21879d" (UID: "c641b326-4164-438e-83fd-5a0b2b21879d"). InnerVolumeSpecName "kube-api-access-xgd9f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 01:18:52.083940 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:18:52.083916    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-f9f50d/rate-test-model-f9f50d-kserve-79b8d64ccf-rch5r"
Jul 16 01:18:52.173235 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:18:52.173201    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c641b326-4164-438e-83fd-5a0b2b21879d-tls-certs\") on node \"ip-10-0-143-65.ec2.internal\" DevicePath \"\""
Jul 16 01:18:52.173235 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:18:52.173231    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c641b326-4164-438e-83fd-5a0b2b21879d-dshm\") on node \"ip-10-0-143-65.ec2.internal\" DevicePath \"\""
Jul 16 01:18:52.173235 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:18:52.173240    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgd9f\" (UniqueName: \"kubernetes.io/projected/c641b326-4164-438e-83fd-5a0b2b21879d-kube-api-access-xgd9f\") on node \"ip-10-0-143-65.ec2.internal\" DevicePath \"\""
Jul 16 01:18:52.505716 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:18:52.505681    2563 generic.go:356] "Generic (PLEG): container finished" podID="c641b326-4164-438e-83fd-5a0b2b21879d" containerID="c0bfc34aa3f12dc1efc2931e33b482ceba258e733b25c3a80bb2c1082a78e0d7" exitCode=0
Jul 16 01:18:52.505984 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:18:52.505725    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-628c71/sub-test-model-628c71-kserve-67988bfbf-qns56"
Jul 16 01:18:52.505984 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:18:52.505756    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-628c71/sub-test-model-628c71-kserve-67988bfbf-qns56" event={"ID":"c641b326-4164-438e-83fd-5a0b2b21879d","Type":"ContainerDied","Data":"c0bfc34aa3f12dc1efc2931e33b482ceba258e733b25c3a80bb2c1082a78e0d7"}
Jul 16 01:18:52.505984 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:18:52.505795    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-628c71/sub-test-model-628c71-kserve-67988bfbf-qns56" event={"ID":"c641b326-4164-438e-83fd-5a0b2b21879d","Type":"ContainerDied","Data":"ee8d7f4a35b0b06490b36d42cb407d5bff9814e89d5153787ded21ad2a0f4c40"}
Jul 16 01:18:52.505984 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:18:52.505817    2563 scope.go:117] "RemoveContainer" containerID="c0bfc34aa3f12dc1efc2931e33b482ceba258e733b25c3a80bb2c1082a78e0d7"
Jul 16 01:18:52.517036 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:18:52.517009    2563 scope.go:117] "RemoveContainer" containerID="c0bfc34aa3f12dc1efc2931e33b482ceba258e733b25c3a80bb2c1082a78e0d7"
Jul 16 01:18:52.517372 ip-10-0-143-65 kubenswrapper[2563]: E0716 01:18:52.517351    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0bfc34aa3f12dc1efc2931e33b482ceba258e733b25c3a80bb2c1082a78e0d7\": container with ID starting with c0bfc34aa3f12dc1efc2931e33b482ceba258e733b25c3a80bb2c1082a78e0d7 not found: ID does not exist" containerID="c0bfc34aa3f12dc1efc2931e33b482ceba258e733b25c3a80bb2c1082a78e0d7"
Jul 16 01:18:52.517444 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:18:52.517379    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0bfc34aa3f12dc1efc2931e33b482ceba258e733b25c3a80bb2c1082a78e0d7"} err="failed to get container status \"c0bfc34aa3f12dc1efc2931e33b482ceba258e733b25c3a80bb2c1082a78e0d7\": rpc error: code = NotFound desc = could not find container \"c0bfc34aa3f12dc1efc2931e33b482ceba258e733b25c3a80bb2c1082a78e0d7\": container with ID starting with c0bfc34aa3f12dc1efc2931e33b482ceba258e733b25c3a80bb2c1082a78e0d7 not found: ID does not exist"
Jul 16 01:18:52.521928 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:18:52.521903    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-f9f50d/rate-test-model-f9f50d-kserve-79b8d64ccf-rch5r"
Jul 16 01:18:52.528615 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:18:52.528586    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-628c71/sub-test-model-628c71-kserve-67988bfbf-qns56"]
Jul 16 01:18:52.530408 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:18:52.530383    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-628c71/sub-test-model-628c71-kserve-67988bfbf-qns56"]
Jul 16 01:18:54.006359 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:18:54.006325    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c641b326-4164-438e-83fd-5a0b2b21879d" path="/var/lib/kubelet/pods/c641b326-4164-438e-83fd-5a0b2b21879d/volumes"
Jul 16 01:18:58.349969 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:18:58.349931    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-628c71/rate-test-model-628c71-kserve-9f69474d7-fgb76"]
Jul 16 01:18:58.350478 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:18:58.350458    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4dda17f-9fe1-4aee-a1b8-003c5be77ba0" containerName="main"
Jul 16 01:18:58.350553 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:18:58.350481    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4dda17f-9fe1-4aee-a1b8-003c5be77ba0" containerName="main"
Jul 16 01:18:58.350553 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:18:58.350508    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c641b326-4164-438e-83fd-5a0b2b21879d" containerName="main"
Jul 16 01:18:58.350553 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:18:58.350518    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c641b326-4164-438e-83fd-5a0b2b21879d" containerName="main"
Jul 16 01:18:58.350706 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:18:58.350617    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4dda17f-9fe1-4aee-a1b8-003c5be77ba0" containerName="main"
Jul 16 01:18:58.350706 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:18:58.350637    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c641b326-4164-438e-83fd-5a0b2b21879d" containerName="main"
Jul 16 01:18:58.355011 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:18:58.354992    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-628c71/rate-test-model-628c71-kserve-9f69474d7-fgb76"
Jul 16 01:18:58.357048 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:18:58.357007    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-628c71\"/\"openshift-service-ca.crt\""
Jul 16 01:18:58.357535 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:18:58.357516    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-628c71\"/\"default-dockercfg-f9hx6\""
Jul 16 01:18:58.357633 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:18:58.357539    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-628c71\"/\"kube-root-ca.crt\""
Jul 16 01:18:58.357633 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:18:58.357539    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-628c71\"/\"rate-test-model-628c71-kserve-self-signed-certs\""
Jul 16 01:18:58.360993 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:18:58.360973    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-628c71/rate-test-model-628c71-kserve-9f69474d7-fgb76"]
Jul 16 01:18:58.429241 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:18:58.429213    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e81ddc9-24cb-4971-af61-0bb4d7d0f387-tmp-dir\") pod \"rate-test-model-628c71-kserve-9f69474d7-fgb76\" (UID: \"4e81ddc9-24cb-4971-af61-0bb4d7d0f387\") " pod="ai-tenant-e2e-shared-b-628c71/rate-test-model-628c71-kserve-9f69474d7-fgb76"
Jul 16 01:18:58.429400 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:18:58.429265    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e81ddc9-24cb-4971-af61-0bb4d7d0f387-tls-certs\") pod \"rate-test-model-628c71-kserve-9f69474d7-fgb76\" (UID: \"4e81ddc9-24cb-4971-af61-0bb4d7d0f387\") " pod="ai-tenant-e2e-shared-b-628c71/rate-test-model-628c71-kserve-9f69474d7-fgb76"
Jul 16 01:18:58.429400 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:18:58.429291    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4b2q\" (UniqueName: \"kubernetes.io/projected/4e81ddc9-24cb-4971-af61-0bb4d7d0f387-kube-api-access-z4b2q\") pod \"rate-test-model-628c71-kserve-9f69474d7-fgb76\" (UID: \"4e81ddc9-24cb-4971-af61-0bb4d7d0f387\") " pod="ai-tenant-e2e-shared-b-628c71/rate-test-model-628c71-kserve-9f69474d7-fgb76"
Jul 16 01:18:58.429400 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:18:58.429378    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e81ddc9-24cb-4971-af61-0bb4d7d0f387-home\") pod \"rate-test-model-628c71-kserve-9f69474d7-fgb76\" (UID: \"4e81ddc9-24cb-4971-af61-0bb4d7d0f387\") " pod="ai-tenant-e2e-shared-b-628c71/rate-test-model-628c71-kserve-9f69474d7-fgb76"
Jul 16 01:18:58.429534 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:18:58.429408    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e81ddc9-24cb-4971-af61-0bb4d7d0f387-dshm\") pod \"rate-test-model-628c71-kserve-9f69474d7-fgb76\" (UID: \"4e81ddc9-24cb-4971-af61-0bb4d7d0f387\") " pod="ai-tenant-e2e-shared-b-628c71/rate-test-model-628c71-kserve-9f69474d7-fgb76"
Jul 16 01:18:58.429534 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:18:58.429443    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e81ddc9-24cb-4971-af61-0bb4d7d0f387-model-cache\") pod \"rate-test-model-628c71-kserve-9f69474d7-fgb76\" (UID: \"4e81ddc9-24cb-4971-af61-0bb4d7d0f387\") " pod="ai-tenant-e2e-shared-b-628c71/rate-test-model-628c71-kserve-9f69474d7-fgb76"
Jul 16 01:18:58.529943 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:18:58.529913    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e81ddc9-24cb-4971-af61-0bb4d7d0f387-tls-certs\") pod \"rate-test-model-628c71-kserve-9f69474d7-fgb76\" (UID: \"4e81ddc9-24cb-4971-af61-0bb4d7d0f387\") " pod="ai-tenant-e2e-shared-b-628c71/rate-test-model-628c71-kserve-9f69474d7-fgb76"
Jul 16 01:18:58.529943 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:18:58.529956    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4b2q\" (UniqueName: \"kubernetes.io/projected/4e81ddc9-24cb-4971-af61-0bb4d7d0f387-kube-api-access-z4b2q\") pod \"rate-test-model-628c71-kserve-9f69474d7-fgb76\" (UID: \"4e81ddc9-24cb-4971-af61-0bb4d7d0f387\") " pod="ai-tenant-e2e-shared-b-628c71/rate-test-model-628c71-kserve-9f69474d7-fgb76"
Jul 16 01:18:58.530201 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:18:58.529996    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e81ddc9-24cb-4971-af61-0bb4d7d0f387-home\") pod \"rate-test-model-628c71-kserve-9f69474d7-fgb76\" (UID: \"4e81ddc9-24cb-4971-af61-0bb4d7d0f387\") " pod="ai-tenant-e2e-shared-b-628c71/rate-test-model-628c71-kserve-9f69474d7-fgb76"
Jul 16 01:18:58.530201 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:18:58.530016    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e81ddc9-24cb-4971-af61-0bb4d7d0f387-dshm\") pod \"rate-test-model-628c71-kserve-9f69474d7-fgb76\" (UID: \"4e81ddc9-24cb-4971-af61-0bb4d7d0f387\") " pod="ai-tenant-e2e-shared-b-628c71/rate-test-model-628c71-kserve-9f69474d7-fgb76"
Jul 16 01:18:58.530201 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:18:58.530045    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e81ddc9-24cb-4971-af61-0bb4d7d0f387-model-cache\") pod \"rate-test-model-628c71-kserve-9f69474d7-fgb76\" (UID: \"4e81ddc9-24cb-4971-af61-0bb4d7d0f387\") " pod="ai-tenant-e2e-shared-b-628c71/rate-test-model-628c71-kserve-9f69474d7-fgb76"
Jul 16 01:18:58.530364 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:18:58.530209    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e81ddc9-24cb-4971-af61-0bb4d7d0f387-tmp-dir\") pod \"rate-test-model-628c71-kserve-9f69474d7-fgb76\" (UID: \"4e81ddc9-24cb-4971-af61-0bb4d7d0f387\") " pod="ai-tenant-e2e-shared-b-628c71/rate-test-model-628c71-kserve-9f69474d7-fgb76"
Jul 16 01:18:58.530528 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:18:58.530506    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e81ddc9-24cb-4971-af61-0bb4d7d0f387-home\") pod \"rate-test-model-628c71-kserve-9f69474d7-fgb76\" (UID: \"4e81ddc9-24cb-4971-af61-0bb4d7d0f387\") " pod="ai-tenant-e2e-shared-b-628c71/rate-test-model-628c71-kserve-9f69474d7-fgb76"
Jul 16 01:18:58.530624 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:18:58.530576    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e81ddc9-24cb-4971-af61-0bb4d7d0f387-tmp-dir\") pod \"rate-test-model-628c71-kserve-9f69474d7-fgb76\" (UID: \"4e81ddc9-24cb-4971-af61-0bb4d7d0f387\") " pod="ai-tenant-e2e-shared-b-628c71/rate-test-model-628c71-kserve-9f69474d7-fgb76"
Jul 16 01:18:58.530687 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:18:58.530666    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e81ddc9-24cb-4971-af61-0bb4d7d0f387-model-cache\") pod \"rate-test-model-628c71-kserve-9f69474d7-fgb76\" (UID: \"4e81ddc9-24cb-4971-af61-0bb4d7d0f387\") " pod="ai-tenant-e2e-shared-b-628c71/rate-test-model-628c71-kserve-9f69474d7-fgb76"
Jul 16 01:18:58.532608 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:18:58.532586    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e81ddc9-24cb-4971-af61-0bb4d7d0f387-dshm\") pod \"rate-test-model-628c71-kserve-9f69474d7-fgb76\" (UID: \"4e81ddc9-24cb-4971-af61-0bb4d7d0f387\") " pod="ai-tenant-e2e-shared-b-628c71/rate-test-model-628c71-kserve-9f69474d7-fgb76"
Jul 16 01:18:58.532973 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:18:58.532953    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e81ddc9-24cb-4971-af61-0bb4d7d0f387-tls-certs\") pod \"rate-test-model-628c71-kserve-9f69474d7-fgb76\" (UID: \"4e81ddc9-24cb-4971-af61-0bb4d7d0f387\") " pod="ai-tenant-e2e-shared-b-628c71/rate-test-model-628c71-kserve-9f69474d7-fgb76"
Jul 16 01:18:58.539274 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:18:58.539255    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4b2q\" (UniqueName: \"kubernetes.io/projected/4e81ddc9-24cb-4971-af61-0bb4d7d0f387-kube-api-access-z4b2q\") pod \"rate-test-model-628c71-kserve-9f69474d7-fgb76\" (UID: \"4e81ddc9-24cb-4971-af61-0bb4d7d0f387\") " pod="ai-tenant-e2e-shared-b-628c71/rate-test-model-628c71-kserve-9f69474d7-fgb76"
Jul 16 01:18:58.665177 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:18:58.665090    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-628c71/rate-test-model-628c71-kserve-9f69474d7-fgb76"
Jul 16 01:18:58.799253 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:18:58.799225    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-628c71/rate-test-model-628c71-kserve-9f69474d7-fgb76"]
Jul 16 01:18:58.800761 ip-10-0-143-65 kubenswrapper[2563]: W0716 01:18:58.800737    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4e81ddc9_24cb_4971_af61_0bb4d7d0f387.slice/crio-f51f90d88447e1a87d66ebc7e8d75ea537d1975d212cf80ccab51ab6d88f37e3 WatchSource:0}: Error finding container f51f90d88447e1a87d66ebc7e8d75ea537d1975d212cf80ccab51ab6d88f37e3: Status 404 returned error can't find the container with id f51f90d88447e1a87d66ebc7e8d75ea537d1975d212cf80ccab51ab6d88f37e3
Jul 16 01:18:59.443937 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:18:59.443900    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl7fx"]
Jul 16 01:18:59.542137 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:18:59.542107    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-628c71/rate-test-model-628c71-kserve-9f69474d7-fgb76" event={"ID":"4e81ddc9-24cb-4971-af61-0bb4d7d0f387","Type":"ContainerStarted","Data":"bcd903a8e5b25341751d78d847654e08282bb9fa9aa4043fc18231128cc37b7c"}
Jul 16 01:18:59.542271 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:18:59.542146    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-628c71/rate-test-model-628c71-kserve-9f69474d7-fgb76" event={"ID":"4e81ddc9-24cb-4971-af61-0bb4d7d0f387","Type":"ContainerStarted","Data":"f51f90d88447e1a87d66ebc7e8d75ea537d1975d212cf80ccab51ab6d88f37e3"}
Jul 16 01:18:59.559602 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:18:59.559534    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-628c71/rate-test-model-628c71-kserve-9f69474d7-fgb76" podStartSLOduration=1.5595166059999999 podStartE2EDuration="1.559516606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 01:18:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 01:18:59.558512405 +0000 UTC m=+2311.993774155" watchObservedRunningTime="2026-07-16 01:18:59.559516606 +0000 UTC m=+2311.994778350"
Jul 16 01:19:08.666285 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:19:08.666250    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-628c71/rate-test-model-628c71-kserve-9f69474d7-fgb76"
Jul 16 01:19:08.666778 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:19:08.666310    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-628c71/rate-test-model-628c71-kserve-9f69474d7-fgb76"
Jul 16 01:19:08.680042 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:19:08.680015    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-628c71/rate-test-model-628c71-kserve-9f69474d7-fgb76"
Jul 16 01:19:09.600777 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:19:09.600740    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-628c71/rate-test-model-628c71-kserve-9f69474d7-fgb76"
Jul 16 01:19:15.849420 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:19:15.849386    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl7fx"]
Jul 16 01:19:30.351575 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:19:30.351539    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl7fx"]
Jul 16 01:19:52.743705 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:19:52.743666    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl7fx"]
Jul 16 01:19:57.280304 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:19:57.280271    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-f9f50d/rate-test-model-f9f50d-kserve-79b8d64ccf-rch5r"]
Jul 16 01:19:58.143315 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:19:58.143278    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-628c71/rate-test-model-628c71-kserve-9f69474d7-fgb76"]
Jul 16 01:20:02.412331 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:02.412290    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7507-139912868546352"]
Jul 16 01:20:02.416090 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:02.416067    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7507-139912868546352"
Jul 16 01:20:02.417973 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:02.417955    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-d9hgd\""
Jul 16 01:20:02.421265 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:02.421245    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7507-139912868546352"]
Jul 16 01:20:02.524947 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:02.524916    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9px9g\" (UniqueName: \"kubernetes.io/projected/2e6cf1c2-51bb-4902-84a7-d6cbdbec74ff-kube-api-access-9px9g\") pod \"test-curl-7507-139912868546352\" (UID: \"2e6cf1c2-51bb-4902-84a7-d6cbdbec74ff\") " pod="opendatahub/test-curl-7507-139912868546352"
Jul 16 01:20:02.625696 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:02.625658    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9px9g\" (UniqueName: \"kubernetes.io/projected/2e6cf1c2-51bb-4902-84a7-d6cbdbec74ff-kube-api-access-9px9g\") pod \"test-curl-7507-139912868546352\" (UID: \"2e6cf1c2-51bb-4902-84a7-d6cbdbec74ff\") " pod="opendatahub/test-curl-7507-139912868546352"
Jul 16 01:20:02.634371 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:02.634324    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9px9g\" (UniqueName: \"kubernetes.io/projected/2e6cf1c2-51bb-4902-84a7-d6cbdbec74ff-kube-api-access-9px9g\") pod \"test-curl-7507-139912868546352\" (UID: \"2e6cf1c2-51bb-4902-84a7-d6cbdbec74ff\") " pod="opendatahub/test-curl-7507-139912868546352"
Jul 16 01:20:02.725560 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:02.725480    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7507-139912868546352"
Jul 16 01:20:02.863772 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:02.863718    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7507-139912868546352"]
Jul 16 01:20:02.868497 ip-10-0-143-65 kubenswrapper[2563]: W0716 01:20:02.868471    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2e6cf1c2_51bb_4902_84a7_d6cbdbec74ff.slice/crio-f011416dfcf4b80f0e653bb0e42d95dca01b85de9d91202d884473ccc3289a63 WatchSource:0}: Error finding container f011416dfcf4b80f0e653bb0e42d95dca01b85de9d91202d884473ccc3289a63: Status 404 returned error can't find the container with id f011416dfcf4b80f0e653bb0e42d95dca01b85de9d91202d884473ccc3289a63
Jul 16 01:20:02.869719 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:02.869702    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 16 01:20:03.856569 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:03.856532    2563 generic.go:356] "Generic (PLEG): container finished" podID="2e6cf1c2-51bb-4902-84a7-d6cbdbec74ff" containerID="5d39a00a6c16f9c7afc95f3df2b729dd13ad0728c7faec736b13c45f6f9afa53" exitCode=0
Jul 16 01:20:03.857022 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:03.856588    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7507-139912868546352" event={"ID":"2e6cf1c2-51bb-4902-84a7-d6cbdbec74ff","Type":"ContainerDied","Data":"5d39a00a6c16f9c7afc95f3df2b729dd13ad0728c7faec736b13c45f6f9afa53"}
Jul 16 01:20:03.857022 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:03.856637    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7507-139912868546352" event={"ID":"2e6cf1c2-51bb-4902-84a7-d6cbdbec74ff","Type":"ContainerStarted","Data":"f011416dfcf4b80f0e653bb0e42d95dca01b85de9d91202d884473ccc3289a63"}
Jul 16 01:20:04.994797 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:04.994774    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7507-139912868546352"
Jul 16 01:20:05.027938 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:05.027911    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7507-139912868546352_2e6cf1c2-51bb-4902-84a7-d6cbdbec74ff/test-curl-7507-139912868546352/0.log"
Jul 16 01:20:05.048806 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:05.048781    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7507-139912868546352"]
Jul 16 01:20:05.051911 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:05.051887    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9px9g\" (UniqueName: \"kubernetes.io/projected/2e6cf1c2-51bb-4902-84a7-d6cbdbec74ff-kube-api-access-9px9g\") pod \"2e6cf1c2-51bb-4902-84a7-d6cbdbec74ff\" (UID: \"2e6cf1c2-51bb-4902-84a7-d6cbdbec74ff\") "
Jul 16 01:20:05.052568 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:05.052551    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7507-139912868546352"]
Jul 16 01:20:05.054029 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:05.054005    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e6cf1c2-51bb-4902-84a7-d6cbdbec74ff-kube-api-access-9px9g" (OuterVolumeSpecName: "kube-api-access-9px9g") pod "2e6cf1c2-51bb-4902-84a7-d6cbdbec74ff" (UID: "2e6cf1c2-51bb-4902-84a7-d6cbdbec74ff"). InnerVolumeSpecName "kube-api-access-9px9g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 01:20:05.152661 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:05.152582    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9px9g\" (UniqueName: \"kubernetes.io/projected/2e6cf1c2-51bb-4902-84a7-d6cbdbec74ff-kube-api-access-9px9g\") on node \"ip-10-0-143-65.ec2.internal\" DevicePath \"\""
Jul 16 01:20:05.219488 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:05.219457    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7507-139912867282064"]
Jul 16 01:20:05.219946 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:05.219927    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e6cf1c2-51bb-4902-84a7-d6cbdbec74ff" containerName="test-curl-7507-139912868546352"
Jul 16 01:20:05.220045 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:05.219947    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e6cf1c2-51bb-4902-84a7-d6cbdbec74ff" containerName="test-curl-7507-139912868546352"
Jul 16 01:20:05.220045 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:05.220027    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e6cf1c2-51bb-4902-84a7-d6cbdbec74ff" containerName="test-curl-7507-139912868546352"
Jul 16 01:20:05.223039 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:05.223021    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7507-139912867282064"
Jul 16 01:20:05.229960 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:05.229933    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7507-139912867282064"]
Jul 16 01:20:05.253335 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:05.253301    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ff8gc\" (UniqueName: \"kubernetes.io/projected/0b3192af-d322-4104-aa5b-f812fbb194d9-kube-api-access-ff8gc\") pod \"test-curl-7507-139912867282064\" (UID: \"0b3192af-d322-4104-aa5b-f812fbb194d9\") " pod="opendatahub/test-curl-7507-139912867282064"
Jul 16 01:20:05.354368 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:05.354334    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ff8gc\" (UniqueName: \"kubernetes.io/projected/0b3192af-d322-4104-aa5b-f812fbb194d9-kube-api-access-ff8gc\") pod \"test-curl-7507-139912867282064\" (UID: \"0b3192af-d322-4104-aa5b-f812fbb194d9\") " pod="opendatahub/test-curl-7507-139912867282064"
Jul 16 01:20:05.362154 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:05.362132    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ff8gc\" (UniqueName: \"kubernetes.io/projected/0b3192af-d322-4104-aa5b-f812fbb194d9-kube-api-access-ff8gc\") pod \"test-curl-7507-139912867282064\" (UID: \"0b3192af-d322-4104-aa5b-f812fbb194d9\") " pod="opendatahub/test-curl-7507-139912867282064"
Jul 16 01:20:05.533017 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:05.532929    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7507-139912867282064"
Jul 16 01:20:05.680109 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:05.680080    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7507-139912867282064"]
Jul 16 01:20:05.691212 ip-10-0-143-65 kubenswrapper[2563]: W0716 01:20:05.691185    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0b3192af_d322_4104_aa5b_f812fbb194d9.slice/crio-3fb0806c725fb03c8e076f55b3f7f2b1a64d5862187447947e2f489207f20c4d WatchSource:0}: Error finding container 3fb0806c725fb03c8e076f55b3f7f2b1a64d5862187447947e2f489207f20c4d: Status 404 returned error can't find the container with id 3fb0806c725fb03c8e076f55b3f7f2b1a64d5862187447947e2f489207f20c4d
Jul 16 01:20:05.867141 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:05.867106    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7507-139912867282064" event={"ID":"0b3192af-d322-4104-aa5b-f812fbb194d9","Type":"ContainerStarted","Data":"3fb0806c725fb03c8e076f55b3f7f2b1a64d5862187447947e2f489207f20c4d"}
Jul 16 01:20:05.868194 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:05.868175    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f011416dfcf4b80f0e653bb0e42d95dca01b85de9d91202d884473ccc3289a63"
Jul 16 01:20:05.868194 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:05.868188    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7507-139912868546352"
Jul 16 01:20:05.878051 ip-10-0-143-65 kubenswrapper[2563]: E0716 01:20:05.878025    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7507-139912868546352\" is forbidden: User \"system:node:ip-10-0-143-65.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-65.ec2.internal' and this object" podUID="2e6cf1c2-51bb-4902-84a7-d6cbdbec74ff" pod="opendatahub/test-curl-7507-139912868546352"
Jul 16 01:20:06.007403 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:06.007362    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e6cf1c2-51bb-4902-84a7-d6cbdbec74ff" path="/var/lib/kubelet/pods/2e6cf1c2-51bb-4902-84a7-d6cbdbec74ff/volumes"
Jul 16 01:20:06.875021 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:06.874987    2563 generic.go:356] "Generic (PLEG): container finished" podID="0b3192af-d322-4104-aa5b-f812fbb194d9" containerID="6eaf2a0cd641612e0b659805dfeb4bdf830eb8f4d94c3762ac9d13b6a6ed263f" exitCode=0
Jul 16 01:20:06.875201 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:06.875048    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7507-139912867282064" event={"ID":"0b3192af-d322-4104-aa5b-f812fbb194d9","Type":"ContainerDied","Data":"6eaf2a0cd641612e0b659805dfeb4bdf830eb8f4d94c3762ac9d13b6a6ed263f"}
Jul 16 01:20:08.006325 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:08.006298    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7507-139912867282064"
Jul 16 01:20:08.026903 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:08.026834    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7507-139912867282064_0b3192af-d322-4104-aa5b-f812fbb194d9/test-curl-7507-139912867282064/0.log"
Jul 16 01:20:08.055424 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:08.055394    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7507-139912867282064"]
Jul 16 01:20:08.061386 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:08.061363    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7507-139912867282064"]
Jul 16 01:20:08.080047 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:08.080020    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ff8gc\" (UniqueName: \"kubernetes.io/projected/0b3192af-d322-4104-aa5b-f812fbb194d9-kube-api-access-ff8gc\") pod \"0b3192af-d322-4104-aa5b-f812fbb194d9\" (UID: \"0b3192af-d322-4104-aa5b-f812fbb194d9\") "
Jul 16 01:20:08.082079 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:08.082055    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b3192af-d322-4104-aa5b-f812fbb194d9-kube-api-access-ff8gc" (OuterVolumeSpecName: "kube-api-access-ff8gc") pod "0b3192af-d322-4104-aa5b-f812fbb194d9" (UID: "0b3192af-d322-4104-aa5b-f812fbb194d9"). InnerVolumeSpecName "kube-api-access-ff8gc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 01:20:08.181327 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:08.181251    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ff8gc\" (UniqueName: \"kubernetes.io/projected/0b3192af-d322-4104-aa5b-f812fbb194d9-kube-api-access-ff8gc\") on node \"ip-10-0-143-65.ec2.internal\" DevicePath \"\""
Jul 16 01:20:08.185996 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:08.185971    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7507-139912867282064"]
Jul 16 01:20:08.186327 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:08.186315    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b3192af-d322-4104-aa5b-f812fbb194d9" containerName="test-curl-7507-139912867282064"
Jul 16 01:20:08.186366 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:08.186328    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b3192af-d322-4104-aa5b-f812fbb194d9" containerName="test-curl-7507-139912867282064"
Jul 16 01:20:08.186417 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:08.186407    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b3192af-d322-4104-aa5b-f812fbb194d9" containerName="test-curl-7507-139912867282064"
Jul 16 01:20:08.189363 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:08.189346    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7507-139912867282064"
Jul 16 01:20:08.195432 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:08.195396    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7507-139912867282064"]
Jul 16 01:20:08.282718 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:08.282670    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nv7ts\" (UniqueName: \"kubernetes.io/projected/6ad69c66-7b9c-45f0-b644-7cb1ec7279fc-kube-api-access-nv7ts\") pod \"test-curl-7507-139912867282064\" (UID: \"6ad69c66-7b9c-45f0-b644-7cb1ec7279fc\") " pod="opendatahub/test-curl-7507-139912867282064"
Jul 16 01:20:08.384075 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:08.384039    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nv7ts\" (UniqueName: \"kubernetes.io/projected/6ad69c66-7b9c-45f0-b644-7cb1ec7279fc-kube-api-access-nv7ts\") pod \"test-curl-7507-139912867282064\" (UID: \"6ad69c66-7b9c-45f0-b644-7cb1ec7279fc\") " pod="opendatahub/test-curl-7507-139912867282064"
Jul 16 01:20:08.391212 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:08.391191    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nv7ts\" (UniqueName: \"kubernetes.io/projected/6ad69c66-7b9c-45f0-b644-7cb1ec7279fc-kube-api-access-nv7ts\") pod \"test-curl-7507-139912867282064\" (UID: \"6ad69c66-7b9c-45f0-b644-7cb1ec7279fc\") " pod="opendatahub/test-curl-7507-139912867282064"
Jul 16 01:20:08.499246 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:08.499151    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7507-139912867282064"
Jul 16 01:20:08.656174 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:08.656139    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7507-139912867282064"]
Jul 16 01:20:08.755349 ip-10-0-143-65 kubenswrapper[2563]: W0716 01:20:08.755320    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6ad69c66_7b9c_45f0_b644_7cb1ec7279fc.slice/crio-ad9892e9b65e01a8eb7b207368cdd5ceb2d17167e90ecf0872cc0a0e31b008cf WatchSource:0}: Error finding container ad9892e9b65e01a8eb7b207368cdd5ceb2d17167e90ecf0872cc0a0e31b008cf: Status 404 returned error can't find the container with id ad9892e9b65e01a8eb7b207368cdd5ceb2d17167e90ecf0872cc0a0e31b008cf
Jul 16 01:20:08.884489 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:08.884457    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7507-139912867282064"
Jul 16 01:20:08.884637 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:08.884500    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3fb0806c725fb03c8e076f55b3f7f2b1a64d5862187447947e2f489207f20c4d"
Jul 16 01:20:08.885592 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:08.885561    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7507-139912867282064" event={"ID":"6ad69c66-7b9c-45f0-b644-7cb1ec7279fc","Type":"ContainerStarted","Data":"ad9892e9b65e01a8eb7b207368cdd5ceb2d17167e90ecf0872cc0a0e31b008cf"}
Jul 16 01:20:08.895848 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:08.895820    2563 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7507-139912867282064" oldPodUID="0b3192af-d322-4104-aa5b-f812fbb194d9" podUID="6ad69c66-7b9c-45f0-b644-7cb1ec7279fc"
Jul 16 01:20:09.892120 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:09.892089    2563 generic.go:356] "Generic (PLEG): container finished" podID="6ad69c66-7b9c-45f0-b644-7cb1ec7279fc" containerID="da0d92f7e661616b9559d9c9778e6e7a959ce8fde8cfd46fc12b74b47d9cc406" exitCode=0
Jul 16 01:20:09.892503 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:09.892168    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7507-139912867282064" event={"ID":"6ad69c66-7b9c-45f0-b644-7cb1ec7279fc","Type":"ContainerDied","Data":"da0d92f7e661616b9559d9c9778e6e7a959ce8fde8cfd46fc12b74b47d9cc406"}
Jul 16 01:20:10.006228 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:10.006193    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b3192af-d322-4104-aa5b-f812fbb194d9" path="/var/lib/kubelet/pods/0b3192af-d322-4104-aa5b-f812fbb194d9/volumes"
Jul 16 01:20:11.026673 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:11.026649    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7507-139912867282064"
Jul 16 01:20:11.048114 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:11.048089    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7507-139912867282064_6ad69c66-7b9c-45f0-b644-7cb1ec7279fc/test-curl-7507-139912867282064/0.log"
Jul 16 01:20:11.072570 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:11.072539    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7507-139912867282064"]
Jul 16 01:20:11.076737 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:11.076702    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7507-139912867282064"]
Jul 16 01:20:11.108291 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:11.108263    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nv7ts\" (UniqueName: \"kubernetes.io/projected/6ad69c66-7b9c-45f0-b644-7cb1ec7279fc-kube-api-access-nv7ts\") pod \"6ad69c66-7b9c-45f0-b644-7cb1ec7279fc\" (UID: \"6ad69c66-7b9c-45f0-b644-7cb1ec7279fc\") "
Jul 16 01:20:11.110291 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:11.110267    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ad69c66-7b9c-45f0-b644-7cb1ec7279fc-kube-api-access-nv7ts" (OuterVolumeSpecName: "kube-api-access-nv7ts") pod "6ad69c66-7b9c-45f0-b644-7cb1ec7279fc" (UID: "6ad69c66-7b9c-45f0-b644-7cb1ec7279fc"). InnerVolumeSpecName "kube-api-access-nv7ts". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 01:20:11.200814 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:11.200738    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7507-139912867281200"]
Jul 16 01:20:11.201121 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:11.201109    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ad69c66-7b9c-45f0-b644-7cb1ec7279fc" containerName="test-curl-7507-139912867282064"
Jul 16 01:20:11.201173 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:11.201123    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ad69c66-7b9c-45f0-b644-7cb1ec7279fc" containerName="test-curl-7507-139912867282064"
Jul 16 01:20:11.201209 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:11.201204    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ad69c66-7b9c-45f0-b644-7cb1ec7279fc" containerName="test-curl-7507-139912867282064"
Jul 16 01:20:11.204116 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:11.204090    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7507-139912867281200"
Jul 16 01:20:11.209329 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:11.209306    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nv7ts\" (UniqueName: \"kubernetes.io/projected/6ad69c66-7b9c-45f0-b644-7cb1ec7279fc-kube-api-access-nv7ts\") on node \"ip-10-0-143-65.ec2.internal\" DevicePath \"\""
Jul 16 01:20:11.209788 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:11.209770    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7507-139912867281200"]
Jul 16 01:20:11.309984 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:11.309951    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gx2r5\" (UniqueName: \"kubernetes.io/projected/9adcea13-9fc9-442b-8897-046ddb53063d-kube-api-access-gx2r5\") pod \"test-curl-7507-139912867281200\" (UID: \"9adcea13-9fc9-442b-8897-046ddb53063d\") " pod="opendatahub/test-curl-7507-139912867281200"
Jul 16 01:20:11.344018 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:11.343987    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl7fx"]
Jul 16 01:20:11.411022 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:11.410990    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gx2r5\" (UniqueName: \"kubernetes.io/projected/9adcea13-9fc9-442b-8897-046ddb53063d-kube-api-access-gx2r5\") pod \"test-curl-7507-139912867281200\" (UID: \"9adcea13-9fc9-442b-8897-046ddb53063d\") " pod="opendatahub/test-curl-7507-139912867281200"
Jul 16 01:20:11.419026 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:11.419000    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gx2r5\" (UniqueName: \"kubernetes.io/projected/9adcea13-9fc9-442b-8897-046ddb53063d-kube-api-access-gx2r5\") pod \"test-curl-7507-139912867281200\" (UID: \"9adcea13-9fc9-442b-8897-046ddb53063d\") " pod="opendatahub/test-curl-7507-139912867281200"
Jul 16 01:20:11.513429 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:11.513349    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7507-139912867281200"
Jul 16 01:20:11.663629 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:11.663593    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7507-139912867281200"]
Jul 16 01:20:11.671201 ip-10-0-143-65 kubenswrapper[2563]: W0716 01:20:11.671174    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9adcea13_9fc9_442b_8897_046ddb53063d.slice/crio-132b78d66c22d54df1513e944d011c99709ad56aa2e3c5569e10d2d1662a33df WatchSource:0}: Error finding container 132b78d66c22d54df1513e944d011c99709ad56aa2e3c5569e10d2d1662a33df: Status 404 returned error can't find the container with id 132b78d66c22d54df1513e944d011c99709ad56aa2e3c5569e10d2d1662a33df
Jul 16 01:20:11.903052 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:11.903018    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7507-139912867282064"
Jul 16 01:20:11.903052 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:11.903044    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ad9892e9b65e01a8eb7b207368cdd5ceb2d17167e90ecf0872cc0a0e31b008cf"
Jul 16 01:20:11.904231 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:11.904207    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7507-139912867281200" event={"ID":"9adcea13-9fc9-442b-8897-046ddb53063d","Type":"ContainerStarted","Data":"132b78d66c22d54df1513e944d011c99709ad56aa2e3c5569e10d2d1662a33df"}
Jul 16 01:20:11.913509 ip-10-0-143-65 kubenswrapper[2563]: E0716 01:20:11.913484    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7507-139912867282064\" is forbidden: User \"system:node:ip-10-0-143-65.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-65.ec2.internal' and this object" podUID="6ad69c66-7b9c-45f0-b644-7cb1ec7279fc" pod="opendatahub/test-curl-7507-139912867282064"
Jul 16 01:20:12.006697 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:12.006665    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ad69c66-7b9c-45f0-b644-7cb1ec7279fc" path="/var/lib/kubelet/pods/6ad69c66-7b9c-45f0-b644-7cb1ec7279fc/volumes"
Jul 16 01:20:12.840551 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:12.840511    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-f9f50d/rate-test-model-f9f50d-kserve-79b8d64ccf-rch5r" podUID="189327f6-723c-44fe-b770-9c3b049dab41" containerName="main" containerID="cri-o://ad639821c4441e4038d0a820a37eca1ef03e7a24360cfc2b8dce74f9b68d5db0" gracePeriod=45
Jul 16 01:20:12.910233 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:12.910196    2563 generic.go:356] "Generic (PLEG): container finished" podID="9adcea13-9fc9-442b-8897-046ddb53063d" containerID="2a227a6c61fdbbc398069191bfc6e4a00805c25b3a4152016f4929c9c3c6d511" exitCode=0
Jul 16 01:20:12.910377 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:12.910272    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7507-139912867281200" event={"ID":"9adcea13-9fc9-442b-8897-046ddb53063d","Type":"ContainerDied","Data":"2a227a6c61fdbbc398069191bfc6e4a00805c25b3a4152016f4929c9c3c6d511"}
Jul 16 01:20:13.093606 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:13.093554    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-f9f50d/rate-test-model-f9f50d-kserve-79b8d64ccf-rch5r"
Jul 16 01:20:13.126451 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:13.126418    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f4hqz\" (UniqueName: \"kubernetes.io/projected/189327f6-723c-44fe-b770-9c3b049dab41-kube-api-access-f4hqz\") pod \"189327f6-723c-44fe-b770-9c3b049dab41\" (UID: \"189327f6-723c-44fe-b770-9c3b049dab41\") "
Jul 16 01:20:13.126623 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:13.126487    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/189327f6-723c-44fe-b770-9c3b049dab41-model-cache\") pod \"189327f6-723c-44fe-b770-9c3b049dab41\" (UID: \"189327f6-723c-44fe-b770-9c3b049dab41\") "
Jul 16 01:20:13.126623 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:13.126505    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/189327f6-723c-44fe-b770-9c3b049dab41-home\") pod \"189327f6-723c-44fe-b770-9c3b049dab41\" (UID: \"189327f6-723c-44fe-b770-9c3b049dab41\") "
Jul 16 01:20:13.126623 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:13.126600    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/189327f6-723c-44fe-b770-9c3b049dab41-dshm\") pod \"189327f6-723c-44fe-b770-9c3b049dab41\" (UID: \"189327f6-723c-44fe-b770-9c3b049dab41\") "
Jul 16 01:20:13.126790 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:13.126704    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/189327f6-723c-44fe-b770-9c3b049dab41-tmp-dir\") pod \"189327f6-723c-44fe-b770-9c3b049dab41\" (UID: \"189327f6-723c-44fe-b770-9c3b049dab41\") "
Jul 16 01:20:13.126790 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:13.126740    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/189327f6-723c-44fe-b770-9c3b049dab41-model-cache" (OuterVolumeSpecName: "model-cache") pod "189327f6-723c-44fe-b770-9c3b049dab41" (UID: "189327f6-723c-44fe-b770-9c3b049dab41"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 01:20:13.126790 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:13.126751    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/189327f6-723c-44fe-b770-9c3b049dab41-home" (OuterVolumeSpecName: "home") pod "189327f6-723c-44fe-b770-9c3b049dab41" (UID: "189327f6-723c-44fe-b770-9c3b049dab41"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 01:20:13.126986 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:13.126791    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/189327f6-723c-44fe-b770-9c3b049dab41-tls-certs\") pod \"189327f6-723c-44fe-b770-9c3b049dab41\" (UID: \"189327f6-723c-44fe-b770-9c3b049dab41\") "
Jul 16 01:20:13.127055 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:13.127026    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/189327f6-723c-44fe-b770-9c3b049dab41-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "189327f6-723c-44fe-b770-9c3b049dab41" (UID: "189327f6-723c-44fe-b770-9c3b049dab41"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 01:20:13.127153 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:13.127126    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/189327f6-723c-44fe-b770-9c3b049dab41-model-cache\") on node \"ip-10-0-143-65.ec2.internal\" DevicePath \"\""
Jul 16 01:20:13.127213 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:13.127156    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/189327f6-723c-44fe-b770-9c3b049dab41-home\") on node \"ip-10-0-143-65.ec2.internal\" DevicePath \"\""
Jul 16 01:20:13.127213 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:13.127166    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/189327f6-723c-44fe-b770-9c3b049dab41-tmp-dir\") on node \"ip-10-0-143-65.ec2.internal\" DevicePath \"\""
Jul 16 01:20:13.128897 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:13.128839    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/189327f6-723c-44fe-b770-9c3b049dab41-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "189327f6-723c-44fe-b770-9c3b049dab41" (UID: "189327f6-723c-44fe-b770-9c3b049dab41"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 01:20:13.129147 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:13.129126    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/189327f6-723c-44fe-b770-9c3b049dab41-dshm" (OuterVolumeSpecName: "dshm") pod "189327f6-723c-44fe-b770-9c3b049dab41" (UID: "189327f6-723c-44fe-b770-9c3b049dab41"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 01:20:13.129200 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:13.129160    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/189327f6-723c-44fe-b770-9c3b049dab41-kube-api-access-f4hqz" (OuterVolumeSpecName: "kube-api-access-f4hqz") pod "189327f6-723c-44fe-b770-9c3b049dab41" (UID: "189327f6-723c-44fe-b770-9c3b049dab41"). InnerVolumeSpecName "kube-api-access-f4hqz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 01:20:13.164051 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:13.163996    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-628c71/rate-test-model-628c71-kserve-9f69474d7-fgb76" podUID="4e81ddc9-24cb-4971-af61-0bb4d7d0f387" containerName="main" containerID="cri-o://bcd903a8e5b25341751d78d847654e08282bb9fa9aa4043fc18231128cc37b7c" gracePeriod=45
Jul 16 01:20:13.228617 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:13.228592    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/189327f6-723c-44fe-b770-9c3b049dab41-tls-certs\") on node \"ip-10-0-143-65.ec2.internal\" DevicePath \"\""
Jul 16 01:20:13.228617 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:13.228615    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4hqz\" (UniqueName: \"kubernetes.io/projected/189327f6-723c-44fe-b770-9c3b049dab41-kube-api-access-f4hqz\") on node \"ip-10-0-143-65.ec2.internal\" DevicePath \"\""
Jul 16 01:20:13.228773 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:13.228624    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/189327f6-723c-44fe-b770-9c3b049dab41-dshm\") on node \"ip-10-0-143-65.ec2.internal\" DevicePath \"\""
Jul 16 01:20:13.406487 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:13.406464    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-628c71/rate-test-model-628c71-kserve-9f69474d7-fgb76"
Jul 16 01:20:13.429992 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:13.429961    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e81ddc9-24cb-4971-af61-0bb4d7d0f387-model-cache\") pod \"4e81ddc9-24cb-4971-af61-0bb4d7d0f387\" (UID: \"4e81ddc9-24cb-4971-af61-0bb4d7d0f387\") "
Jul 16 01:20:13.430163 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:13.430031    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e81ddc9-24cb-4971-af61-0bb4d7d0f387-tmp-dir\") pod \"4e81ddc9-24cb-4971-af61-0bb4d7d0f387\" (UID: \"4e81ddc9-24cb-4971-af61-0bb4d7d0f387\") "
Jul 16 01:20:13.430163 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:13.430155    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e81ddc9-24cb-4971-af61-0bb4d7d0f387-home\") pod \"4e81ddc9-24cb-4971-af61-0bb4d7d0f387\" (UID: \"4e81ddc9-24cb-4971-af61-0bb4d7d0f387\") "
Jul 16 01:20:13.430263 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:13.430193    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e81ddc9-24cb-4971-af61-0bb4d7d0f387-tls-certs\") pod \"4e81ddc9-24cb-4971-af61-0bb4d7d0f387\" (UID: \"4e81ddc9-24cb-4971-af61-0bb4d7d0f387\") "
Jul 16 01:20:13.430263 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:13.430199    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e81ddc9-24cb-4971-af61-0bb4d7d0f387-model-cache" (OuterVolumeSpecName: "model-cache") pod "4e81ddc9-24cb-4971-af61-0bb4d7d0f387" (UID: "4e81ddc9-24cb-4971-af61-0bb4d7d0f387"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 01:20:13.430263 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:13.430223    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z4b2q\" (UniqueName: \"kubernetes.io/projected/4e81ddc9-24cb-4971-af61-0bb4d7d0f387-kube-api-access-z4b2q\") pod \"4e81ddc9-24cb-4971-af61-0bb4d7d0f387\" (UID: \"4e81ddc9-24cb-4971-af61-0bb4d7d0f387\") "
Jul 16 01:20:13.430427 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:13.430262    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e81ddc9-24cb-4971-af61-0bb4d7d0f387-dshm\") pod \"4e81ddc9-24cb-4971-af61-0bb4d7d0f387\" (UID: \"4e81ddc9-24cb-4971-af61-0bb4d7d0f387\") "
Jul 16 01:20:13.430427 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:13.430269    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e81ddc9-24cb-4971-af61-0bb4d7d0f387-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4e81ddc9-24cb-4971-af61-0bb4d7d0f387" (UID: "4e81ddc9-24cb-4971-af61-0bb4d7d0f387"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 01:20:13.430519 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:13.430453    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e81ddc9-24cb-4971-af61-0bb4d7d0f387-home" (OuterVolumeSpecName: "home") pod "4e81ddc9-24cb-4971-af61-0bb4d7d0f387" (UID: "4e81ddc9-24cb-4971-af61-0bb4d7d0f387"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 01:20:13.430559 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:13.430541    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e81ddc9-24cb-4971-af61-0bb4d7d0f387-home\") on node \"ip-10-0-143-65.ec2.internal\" DevicePath \"\""
Jul 16 01:20:13.430597 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:13.430560    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e81ddc9-24cb-4971-af61-0bb4d7d0f387-model-cache\") on node \"ip-10-0-143-65.ec2.internal\" DevicePath \"\""
Jul 16 01:20:13.430597 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:13.430574    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e81ddc9-24cb-4971-af61-0bb4d7d0f387-tmp-dir\") on node \"ip-10-0-143-65.ec2.internal\" DevicePath \"\""
Jul 16 01:20:13.432223 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:13.432202    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e81ddc9-24cb-4971-af61-0bb4d7d0f387-dshm" (OuterVolumeSpecName: "dshm") pod "4e81ddc9-24cb-4971-af61-0bb4d7d0f387" (UID: "4e81ddc9-24cb-4971-af61-0bb4d7d0f387"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 01:20:13.432669 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:13.432650    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e81ddc9-24cb-4971-af61-0bb4d7d0f387-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4e81ddc9-24cb-4971-af61-0bb4d7d0f387" (UID: "4e81ddc9-24cb-4971-af61-0bb4d7d0f387"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 01:20:13.432721 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:13.432661    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e81ddc9-24cb-4971-af61-0bb4d7d0f387-kube-api-access-z4b2q" (OuterVolumeSpecName: "kube-api-access-z4b2q") pod "4e81ddc9-24cb-4971-af61-0bb4d7d0f387" (UID: "4e81ddc9-24cb-4971-af61-0bb4d7d0f387"). InnerVolumeSpecName "kube-api-access-z4b2q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 01:20:13.531079 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:13.531042    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e81ddc9-24cb-4971-af61-0bb4d7d0f387-tls-certs\") on node \"ip-10-0-143-65.ec2.internal\" DevicePath \"\""
Jul 16 01:20:13.531079 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:13.531072    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4b2q\" (UniqueName: \"kubernetes.io/projected/4e81ddc9-24cb-4971-af61-0bb4d7d0f387-kube-api-access-z4b2q\") on node \"ip-10-0-143-65.ec2.internal\" DevicePath \"\""
Jul 16 01:20:13.531079 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:13.531081    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e81ddc9-24cb-4971-af61-0bb4d7d0f387-dshm\") on node \"ip-10-0-143-65.ec2.internal\" DevicePath \"\""
Jul 16 01:20:13.917075 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:13.917042    2563 generic.go:356] "Generic (PLEG): container finished" podID="4e81ddc9-24cb-4971-af61-0bb4d7d0f387" containerID="bcd903a8e5b25341751d78d847654e08282bb9fa9aa4043fc18231128cc37b7c" exitCode=0
Jul 16 01:20:13.917519 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:13.917088    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-628c71/rate-test-model-628c71-kserve-9f69474d7-fgb76"
Jul 16 01:20:13.917519 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:13.917132    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-628c71/rate-test-model-628c71-kserve-9f69474d7-fgb76" event={"ID":"4e81ddc9-24cb-4971-af61-0bb4d7d0f387","Type":"ContainerDied","Data":"bcd903a8e5b25341751d78d847654e08282bb9fa9aa4043fc18231128cc37b7c"}
Jul 16 01:20:13.917519 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:13.917169    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-628c71/rate-test-model-628c71-kserve-9f69474d7-fgb76" event={"ID":"4e81ddc9-24cb-4971-af61-0bb4d7d0f387","Type":"ContainerDied","Data":"f51f90d88447e1a87d66ebc7e8d75ea537d1975d212cf80ccab51ab6d88f37e3"}
Jul 16 01:20:13.917519 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:13.917189    2563 scope.go:117] "RemoveContainer" containerID="bcd903a8e5b25341751d78d847654e08282bb9fa9aa4043fc18231128cc37b7c"
Jul 16 01:20:13.918578 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:13.918549    2563 generic.go:356] "Generic (PLEG): container finished" podID="189327f6-723c-44fe-b770-9c3b049dab41" containerID="ad639821c4441e4038d0a820a37eca1ef03e7a24360cfc2b8dce74f9b68d5db0" exitCode=0
Jul 16 01:20:13.918699 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:13.918585    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-f9f50d/rate-test-model-f9f50d-kserve-79b8d64ccf-rch5r"
Jul 16 01:20:13.918699 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:13.918594    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-f9f50d/rate-test-model-f9f50d-kserve-79b8d64ccf-rch5r" event={"ID":"189327f6-723c-44fe-b770-9c3b049dab41","Type":"ContainerDied","Data":"ad639821c4441e4038d0a820a37eca1ef03e7a24360cfc2b8dce74f9b68d5db0"}
Jul 16 01:20:13.918699 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:13.918632    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-f9f50d/rate-test-model-f9f50d-kserve-79b8d64ccf-rch5r" event={"ID":"189327f6-723c-44fe-b770-9c3b049dab41","Type":"ContainerDied","Data":"294d6b64bc2afe96019ea4ed462fcd5712539793045b3b1c3145ab12a1ebf574"}
Jul 16 01:20:13.926651 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:13.926630    2563 scope.go:117] "RemoveContainer" containerID="bcd903a8e5b25341751d78d847654e08282bb9fa9aa4043fc18231128cc37b7c"
Jul 16 01:20:13.927008 ip-10-0-143-65 kubenswrapper[2563]: E0716 01:20:13.926983    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bcd903a8e5b25341751d78d847654e08282bb9fa9aa4043fc18231128cc37b7c\": container with ID starting with bcd903a8e5b25341751d78d847654e08282bb9fa9aa4043fc18231128cc37b7c not found: ID does not exist" containerID="bcd903a8e5b25341751d78d847654e08282bb9fa9aa4043fc18231128cc37b7c"
Jul 16 01:20:13.927073 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:13.927017    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bcd903a8e5b25341751d78d847654e08282bb9fa9aa4043fc18231128cc37b7c"} err="failed to get container status \"bcd903a8e5b25341751d78d847654e08282bb9fa9aa4043fc18231128cc37b7c\": rpc error: code = NotFound desc = could not find container \"bcd903a8e5b25341751d78d847654e08282bb9fa9aa4043fc18231128cc37b7c\": container with ID starting with bcd903a8e5b25341751d78d847654e08282bb9fa9aa4043fc18231128cc37b7c not found: ID does not exist"
Jul 16 01:20:13.927073 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:13.927039    2563 scope.go:117] "RemoveContainer" containerID="ad639821c4441e4038d0a820a37eca1ef03e7a24360cfc2b8dce74f9b68d5db0"
Jul 16 01:20:13.935223 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:13.935204    2563 scope.go:117] "RemoveContainer" containerID="ad639821c4441e4038d0a820a37eca1ef03e7a24360cfc2b8dce74f9b68d5db0"
Jul 16 01:20:13.935523 ip-10-0-143-65 kubenswrapper[2563]: E0716 01:20:13.935506    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad639821c4441e4038d0a820a37eca1ef03e7a24360cfc2b8dce74f9b68d5db0\": container with ID starting with ad639821c4441e4038d0a820a37eca1ef03e7a24360cfc2b8dce74f9b68d5db0 not found: ID does not exist" containerID="ad639821c4441e4038d0a820a37eca1ef03e7a24360cfc2b8dce74f9b68d5db0"
Jul 16 01:20:13.935576 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:13.935529    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad639821c4441e4038d0a820a37eca1ef03e7a24360cfc2b8dce74f9b68d5db0"} err="failed to get container status \"ad639821c4441e4038d0a820a37eca1ef03e7a24360cfc2b8dce74f9b68d5db0\": rpc error: code = NotFound desc = could not find container \"ad639821c4441e4038d0a820a37eca1ef03e7a24360cfc2b8dce74f9b68d5db0\": container with ID starting with ad639821c4441e4038d0a820a37eca1ef03e7a24360cfc2b8dce74f9b68d5db0 not found: ID does not exist"
Jul 16 01:20:13.942216 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:13.942193    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-628c71/rate-test-model-628c71-kserve-9f69474d7-fgb76"]
Jul 16 01:20:13.948981 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:13.948961    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-628c71/rate-test-model-628c71-kserve-9f69474d7-fgb76"]
Jul 16 01:20:13.960460 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:13.960436    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-f9f50d/rate-test-model-f9f50d-kserve-79b8d64ccf-rch5r"]
Jul 16 01:20:13.964686 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:13.964665    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-f9f50d/rate-test-model-f9f50d-kserve-79b8d64ccf-rch5r"]
Jul 16 01:20:14.007380 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:14.007346    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="189327f6-723c-44fe-b770-9c3b049dab41" path="/var/lib/kubelet/pods/189327f6-723c-44fe-b770-9c3b049dab41/volumes"
Jul 16 01:20:14.007779 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:14.007763    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e81ddc9-24cb-4971-af61-0bb4d7d0f387" path="/var/lib/kubelet/pods/4e81ddc9-24cb-4971-af61-0bb4d7d0f387/volumes"
Jul 16 01:20:14.052458 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:14.052433    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7507-139912867281200"
Jul 16 01:20:14.072353 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:14.072318    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7507-139912867281200_9adcea13-9fc9-442b-8897-046ddb53063d/test-curl-7507-139912867281200/0.log"
Jul 16 01:20:14.096057 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:14.096036    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7507-139912867281200"]
Jul 16 01:20:14.097725 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:14.097703    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7507-139912867281200"]
Jul 16 01:20:14.140344 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:14.140323    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gx2r5\" (UniqueName: \"kubernetes.io/projected/9adcea13-9fc9-442b-8897-046ddb53063d-kube-api-access-gx2r5\") pod \"9adcea13-9fc9-442b-8897-046ddb53063d\" (UID: \"9adcea13-9fc9-442b-8897-046ddb53063d\") "
Jul 16 01:20:14.142280 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:14.142258    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9adcea13-9fc9-442b-8897-046ddb53063d-kube-api-access-gx2r5" (OuterVolumeSpecName: "kube-api-access-gx2r5") pod "9adcea13-9fc9-442b-8897-046ddb53063d" (UID: "9adcea13-9fc9-442b-8897-046ddb53063d"). InnerVolumeSpecName "kube-api-access-gx2r5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 01:20:14.222894 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:14.222791    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7507-139912867281200"]
Jul 16 01:20:14.223156 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:14.223143    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="189327f6-723c-44fe-b770-9c3b049dab41" containerName="main"
Jul 16 01:20:14.223200 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:14.223157    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="189327f6-723c-44fe-b770-9c3b049dab41" containerName="main"
Jul 16 01:20:14.223200 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:14.223165    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e81ddc9-24cb-4971-af61-0bb4d7d0f387" containerName="main"
Jul 16 01:20:14.223200 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:14.223171    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e81ddc9-24cb-4971-af61-0bb4d7d0f387" containerName="main"
Jul 16 01:20:14.223200 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:14.223182    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9adcea13-9fc9-442b-8897-046ddb53063d" containerName="test-curl-7507-139912867281200"
Jul 16 01:20:14.223200 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:14.223189    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="9adcea13-9fc9-442b-8897-046ddb53063d" containerName="test-curl-7507-139912867281200"
Jul 16 01:20:14.223349 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:14.223254    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="189327f6-723c-44fe-b770-9c3b049dab41" containerName="main"
Jul 16 01:20:14.223349 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:14.223262    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="9adcea13-9fc9-442b-8897-046ddb53063d" containerName="test-curl-7507-139912867281200"
Jul 16 01:20:14.223349 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:14.223271    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e81ddc9-24cb-4971-af61-0bb4d7d0f387" containerName="main"
Jul 16 01:20:14.226297 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:14.226279    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7507-139912867281200"
Jul 16 01:20:14.231415 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:14.231390    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7507-139912867281200"]
Jul 16 01:20:14.241965 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:14.241941    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gx2r5\" (UniqueName: \"kubernetes.io/projected/9adcea13-9fc9-442b-8897-046ddb53063d-kube-api-access-gx2r5\") on node \"ip-10-0-143-65.ec2.internal\" DevicePath \"\""
Jul 16 01:20:14.343110 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:14.343078    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9w6q\" (UniqueName: \"kubernetes.io/projected/e83e568c-02ed-4500-99fd-f03a12726c37-kube-api-access-f9w6q\") pod \"test-curl-7507-139912867281200\" (UID: \"e83e568c-02ed-4500-99fd-f03a12726c37\") " pod="opendatahub/test-curl-7507-139912867281200"
Jul 16 01:20:14.444249 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:14.444212    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9w6q\" (UniqueName: \"kubernetes.io/projected/e83e568c-02ed-4500-99fd-f03a12726c37-kube-api-access-f9w6q\") pod \"test-curl-7507-139912867281200\" (UID: \"e83e568c-02ed-4500-99fd-f03a12726c37\") " pod="opendatahub/test-curl-7507-139912867281200"
Jul 16 01:20:14.452621 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:14.452588    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9w6q\" (UniqueName: \"kubernetes.io/projected/e83e568c-02ed-4500-99fd-f03a12726c37-kube-api-access-f9w6q\") pod \"test-curl-7507-139912867281200\" (UID: \"e83e568c-02ed-4500-99fd-f03a12726c37\") " pod="opendatahub/test-curl-7507-139912867281200"
Jul 16 01:20:14.535732 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:14.535645    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7507-139912867281200"
Jul 16 01:20:14.671453 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:14.671418    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7507-139912867281200"]
Jul 16 01:20:14.674492 ip-10-0-143-65 kubenswrapper[2563]: W0716 01:20:14.674465    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode83e568c_02ed_4500_99fd_f03a12726c37.slice/crio-8832fd24370e70e7be9f279bdf4d49f484ca5282d0758899e16ee9ab7e60b04d WatchSource:0}: Error finding container 8832fd24370e70e7be9f279bdf4d49f484ca5282d0758899e16ee9ab7e60b04d: Status 404 returned error can't find the container with id 8832fd24370e70e7be9f279bdf4d49f484ca5282d0758899e16ee9ab7e60b04d
Jul 16 01:20:14.929324 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:14.929290    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7507-139912867281200"
Jul 16 01:20:14.929775 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:14.929295    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="132b78d66c22d54df1513e944d011c99709ad56aa2e3c5569e10d2d1662a33df"
Jul 16 01:20:14.930432 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:14.930407    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7507-139912867281200" event={"ID":"e83e568c-02ed-4500-99fd-f03a12726c37","Type":"ContainerStarted","Data":"8832fd24370e70e7be9f279bdf4d49f484ca5282d0758899e16ee9ab7e60b04d"}
Jul 16 01:20:14.940563 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:14.940538    2563 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7507-139912867281200" oldPodUID="9adcea13-9fc9-442b-8897-046ddb53063d" podUID="e83e568c-02ed-4500-99fd-f03a12726c37"
Jul 16 01:20:15.937782 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:15.937752    2563 generic.go:356] "Generic (PLEG): container finished" podID="e83e568c-02ed-4500-99fd-f03a12726c37" containerID="f247bfff85ddb3691fe751efd45812f8aec8b66f329d46fc197528879e554fee" exitCode=0
Jul 16 01:20:15.938155 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:15.937832    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7507-139912867281200" event={"ID":"e83e568c-02ed-4500-99fd-f03a12726c37","Type":"ContainerDied","Data":"f247bfff85ddb3691fe751efd45812f8aec8b66f329d46fc197528879e554fee"}
Jul 16 01:20:16.012837 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:16.012802    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9adcea13-9fc9-442b-8897-046ddb53063d" path="/var/lib/kubelet/pods/9adcea13-9fc9-442b-8897-046ddb53063d/volumes"
Jul 16 01:20:17.071928 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:17.071902    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7507-139912867281200"
Jul 16 01:20:17.097512 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:17.097485    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7507-139912867281200_e83e568c-02ed-4500-99fd-f03a12726c37/test-curl-7507-139912867281200/0.log"
Jul 16 01:20:17.126567 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:17.126510    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7507-139912867281200"]
Jul 16 01:20:17.129435 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:17.129413    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7507-139912867281200"]
Jul 16 01:20:17.167798 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:17.167766    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f9w6q\" (UniqueName: \"kubernetes.io/projected/e83e568c-02ed-4500-99fd-f03a12726c37-kube-api-access-f9w6q\") pod \"e83e568c-02ed-4500-99fd-f03a12726c37\" (UID: \"e83e568c-02ed-4500-99fd-f03a12726c37\") "
Jul 16 01:20:17.169821 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:17.169800    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e83e568c-02ed-4500-99fd-f03a12726c37-kube-api-access-f9w6q" (OuterVolumeSpecName: "kube-api-access-f9w6q") pod "e83e568c-02ed-4500-99fd-f03a12726c37" (UID: "e83e568c-02ed-4500-99fd-f03a12726c37"). InnerVolumeSpecName "kube-api-access-f9w6q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 01:20:17.268453 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:17.268378    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9w6q\" (UniqueName: \"kubernetes.io/projected/e83e568c-02ed-4500-99fd-f03a12726c37-kube-api-access-f9w6q\") on node \"ip-10-0-143-65.ec2.internal\" DevicePath \"\""
Jul 16 01:20:17.948880 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:17.948847    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7507-139912867281200"
Jul 16 01:20:17.949027 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:17.948896    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8832fd24370e70e7be9f279bdf4d49f484ca5282d0758899e16ee9ab7e60b04d"
Jul 16 01:20:17.958900 ip-10-0-143-65 kubenswrapper[2563]: E0716 01:20:17.958849    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7507-139912867281200\" is forbidden: User \"system:node:ip-10-0-143-65.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-65.ec2.internal' and this object" podUID="e83e568c-02ed-4500-99fd-f03a12726c37" pod="opendatahub/test-curl-7507-139912867281200"
Jul 16 01:20:18.004315 ip-10-0-143-65 kubenswrapper[2563]: E0716 01:20:18.004276    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7507-139912867281200\" is forbidden: User \"system:node:ip-10-0-143-65.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-65.ec2.internal' and this object" podUID="e83e568c-02ed-4500-99fd-f03a12726c37" pod="opendatahub/test-curl-7507-139912867281200"
Jul 16 01:20:18.006244 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:18.006222    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e83e568c-02ed-4500-99fd-f03a12726c37" path="/var/lib/kubelet/pods/e83e568c-02ed-4500-99fd-f03a12726c37/volumes"
Jul 16 01:20:28.080256 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:28.080228    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bn5hr_e990a21d-614b-4d1d-be54-0df6e44487ed/ovn-acl-logging/0.log"
Jul 16 01:20:28.083738 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:28.083719    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bn5hr_e990a21d-614b-4d1d-be54-0df6e44487ed/ovn-acl-logging/0.log"
Jul 16 01:20:28.546165 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:28.546086    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl7fx"]
Jul 16 01:20:32.225133 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:32.225101    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-c6466dcb6-jvh97_d06ed8c7-ae3c-4c21-83bd-fd781e9f23c8/authorino/0.log"
Jul 16 01:20:37.734213 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:37.734179    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-cgjd4_f670d048-05a1-44e0-bb47-f33afeba3880/manager/0.log"
Jul 16 01:20:38.517675 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:38.517647    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5cf8c55d45-8d6hh_c9446ea3-dac9-4a1a-9ad3-1fda85d9f151/copy-manifests/0.log"
Jul 16 01:20:38.530708 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:38.530683    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5cf8c55d45-8d6hh_c9446ea3-dac9-4a1a-9ad3-1fda85d9f151/manager/0.log"
Jul 16 01:20:40.506206 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:40.506175    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29736075-qhm25_1658112c-ac54-462f-badd-ebd7d9c24d14/cleanup/0.log"
Jul 16 01:20:40.839108 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:40.839083    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-e2e-api-b-2814a2-29736075-8sw6g_e494d6b2-80c8-42fb-8678-e40796989e48/cleanup/0.log"
Jul 16 01:20:41.061089 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:41.061060    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-e2e-inf-a-7d5e85-29736075-cvfwd_61ee0ce1-2f13-407a-a3c5-9f67846606dc/cleanup/0.log"
Jul 16 01:20:41.338739 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:41.338702    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hl7fx"]
Jul 16 01:20:43.080733 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:43.080700    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-c6466dcb6-jvh97_d06ed8c7-ae3c-4c21-83bd-fd781e9f23c8/authorino/0.log"
Jul 16 01:20:43.444229 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:43.444155    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-pv7nf_36c1deac-8dd8-4797-a13f-f335d7ddcb12/kuadrant-console-plugin/0.log"
Jul 16 01:20:43.821822 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:43.821791    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-hl7fx_991a6621-30fa-4462-8b75-7acfbd97cb4a/limitador/0.log"
Jul 16 01:20:44.666568 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:44.666539    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7ffc9f7c5d-k7f7b_b98801ba-36e7-4e48-84b8-56709b911b43/kube-auth-proxy/0.log"
Jul 16 01:20:45.952340 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:45.952308    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-z6sx5_b2d0e466-a859-4e50-8703-43adcdcb1aaa/main/0.log"
Jul 16 01:20:46.060116 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:46.060087    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5fc6fb456f-fzg7k_343c8ea6-28e1-44cc-aa55-41d91d40fbe1/main/0.log"
Jul 16 01:20:51.753993 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:51.753904    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-5b5g8/must-gather-kl5l6"]
Jul 16 01:20:51.754438 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:51.754411    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e83e568c-02ed-4500-99fd-f03a12726c37" containerName="test-curl-7507-139912867281200"
Jul 16 01:20:51.754438 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:51.754431    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e83e568c-02ed-4500-99fd-f03a12726c37" containerName="test-curl-7507-139912867281200"
Jul 16 01:20:51.754559 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:51.754543    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e83e568c-02ed-4500-99fd-f03a12726c37" containerName="test-curl-7507-139912867281200"
Jul 16 01:20:51.758087 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:51.758066    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5b5g8/must-gather-kl5l6"
Jul 16 01:20:51.761698 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:51.761677    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-5b5g8\"/\"default-dockercfg-h42dq\""
Jul 16 01:20:51.761837 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:51.761716    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5b5g8\"/\"kube-root-ca.crt\""
Jul 16 01:20:51.762188 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:51.762168    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5b5g8\"/\"openshift-service-ca.crt\""
Jul 16 01:20:51.777367 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:51.777337    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5b5g8/must-gather-kl5l6"]
Jul 16 01:20:51.785284 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:51.785261    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zn2wn\" (UniqueName: \"kubernetes.io/projected/2f7d5bf5-3a53-4ead-b058-e91530731363-kube-api-access-zn2wn\") pod \"must-gather-kl5l6\" (UID: \"2f7d5bf5-3a53-4ead-b058-e91530731363\") " pod="openshift-must-gather-5b5g8/must-gather-kl5l6"
Jul 16 01:20:51.785393 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:51.785353    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/2f7d5bf5-3a53-4ead-b058-e91530731363-must-gather-output\") pod \"must-gather-kl5l6\" (UID: \"2f7d5bf5-3a53-4ead-b058-e91530731363\") " pod="openshift-must-gather-5b5g8/must-gather-kl5l6"
Jul 16 01:20:51.886620 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:51.886583    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zn2wn\" (UniqueName: \"kubernetes.io/projected/2f7d5bf5-3a53-4ead-b058-e91530731363-kube-api-access-zn2wn\") pod \"must-gather-kl5l6\" (UID: \"2f7d5bf5-3a53-4ead-b058-e91530731363\") " pod="openshift-must-gather-5b5g8/must-gather-kl5l6"
Jul 16 01:20:51.886833 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:51.886665    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/2f7d5bf5-3a53-4ead-b058-e91530731363-must-gather-output\") pod \"must-gather-kl5l6\" (UID: \"2f7d5bf5-3a53-4ead-b058-e91530731363\") " pod="openshift-must-gather-5b5g8/must-gather-kl5l6"
Jul 16 01:20:51.886977 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:51.886961    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/2f7d5bf5-3a53-4ead-b058-e91530731363-must-gather-output\") pod \"must-gather-kl5l6\" (UID: \"2f7d5bf5-3a53-4ead-b058-e91530731363\") " pod="openshift-must-gather-5b5g8/must-gather-kl5l6"
Jul 16 01:20:51.897550 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:51.897521    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zn2wn\" (UniqueName: \"kubernetes.io/projected/2f7d5bf5-3a53-4ead-b058-e91530731363-kube-api-access-zn2wn\") pod \"must-gather-kl5l6\" (UID: \"2f7d5bf5-3a53-4ead-b058-e91530731363\") " pod="openshift-must-gather-5b5g8/must-gather-kl5l6"
Jul 16 01:20:52.067131 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:52.067048    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5b5g8/must-gather-kl5l6"
Jul 16 01:20:52.200982 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:52.200957    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5b5g8/must-gather-kl5l6"]
Jul 16 01:20:52.203561 ip-10-0-143-65 kubenswrapper[2563]: W0716 01:20:52.203534    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2f7d5bf5_3a53_4ead_b058_e91530731363.slice/crio-094b030456da60237ac3ec9d582d5c64104237eda948cf0baf55f4ef0fa822f8 WatchSource:0}: Error finding container 094b030456da60237ac3ec9d582d5c64104237eda948cf0baf55f4ef0fa822f8: Status 404 returned error can't find the container with id 094b030456da60237ac3ec9d582d5c64104237eda948cf0baf55f4ef0fa822f8
Jul 16 01:20:53.124456 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:53.124426    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5b5g8/must-gather-kl5l6" event={"ID":"2f7d5bf5-3a53-4ead-b058-e91530731363","Type":"ContainerStarted","Data":"094b030456da60237ac3ec9d582d5c64104237eda948cf0baf55f4ef0fa822f8"}
Jul 16 01:20:54.135294 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:54.135257    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5b5g8/must-gather-kl5l6" event={"ID":"2f7d5bf5-3a53-4ead-b058-e91530731363","Type":"ContainerStarted","Data":"e94ae9fda59f69851c3c938bcfa9f8caee5f1b680fb56dd20e0b7d85195a51c1"}
Jul 16 01:20:54.135294 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:54.135301    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5b5g8/must-gather-kl5l6" event={"ID":"2f7d5bf5-3a53-4ead-b058-e91530731363","Type":"ContainerStarted","Data":"4ae99266854579916f4ed021da922d1d176d117ed0db3a4bd47c61d3fcac1a1d"}
Jul 16 01:20:54.154653 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:54.154596    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-5b5g8/must-gather-kl5l6" podStartSLOduration=2.31920586 podStartE2EDuration="3.154575224s" totalImagesPullingTime="835.369364ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 01:20:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 01:20:54.152675755 +0000 UTC m=+2426.587937493" watchObservedRunningTime="2026-07-16 01:20:54.154575224 +0000 UTC m=+2426.589836964"
Jul 16 01:20:54.693319 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:54.693286    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-hgcmx_16a7fa19-0ce5-49ca-96ef-dde5ae694dc0/global-pull-secret-syncer/0.log"
Jul 16 01:20:54.884298 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:54.884265    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-tm5f5_39f3a671-1715-4dd3-9ca0-0b714dde3281/konnectivity-agent/0.log"
Jul 16 01:20:54.984343 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:20:54.984272    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-143-65.ec2.internal_ebeafacc0368100d06658fd0ec2ae27a/haproxy/0.log"
Jul 16 01:21:00.481293 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:21:00.481063    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-c6466dcb6-jvh97_d06ed8c7-ae3c-4c21-83bd-fd781e9f23c8/authorino/0.log"
Jul 16 01:21:00.580418 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:21:00.580382    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-pv7nf_36c1deac-8dd8-4797-a13f-f335d7ddcb12/kuadrant-console-plugin/0.log"
Jul 16 01:21:00.865003 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:21:00.864972    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-hl7fx_991a6621-30fa-4462-8b75-7acfbd97cb4a/limitador/0.log"
Jul 16 01:21:02.567141 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:21:02.567057    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3c93649a-ae28-4bb7-8452-acf2b9b42acd/alertmanager/0.log"
Jul 16 01:21:02.606586 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:21:02.606557    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3c93649a-ae28-4bb7-8452-acf2b9b42acd/config-reloader/0.log"
Jul 16 01:21:02.690196 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:21:02.690162    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3c93649a-ae28-4bb7-8452-acf2b9b42acd/kube-rbac-proxy-web/0.log"
Jul 16 01:21:02.741486 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:21:02.741462    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3c93649a-ae28-4bb7-8452-acf2b9b42acd/kube-rbac-proxy/0.log"
Jul 16 01:21:02.771032 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:21:02.770991    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3c93649a-ae28-4bb7-8452-acf2b9b42acd/kube-rbac-proxy-metric/0.log"
Jul 16 01:21:02.793967 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:21:02.793938    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3c93649a-ae28-4bb7-8452-acf2b9b42acd/prom-label-proxy/0.log"
Jul 16 01:21:02.820972 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:21:02.820944    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3c93649a-ae28-4bb7-8452-acf2b9b42acd/init-config-reloader/0.log"
Jul 16 01:21:02.983225 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:21:02.983194    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-67fddbd646-n6lrk_151b0be5-d373-4629-89b5-cf407bdab706/metrics-server/0.log"
Jul 16 01:21:03.126912 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:21:03.126818    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gfrth_634869d9-271a-4673-b367-e673b5e7b867/node-exporter/0.log"
Jul 16 01:21:03.159680 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:21:03.159642    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gfrth_634869d9-271a-4673-b367-e673b5e7b867/kube-rbac-proxy/0.log"
Jul 16 01:21:03.182166 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:21:03.182139    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gfrth_634869d9-271a-4673-b367-e673b5e7b867/init-textfile/0.log"
Jul 16 01:21:03.300379 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:21:03.300332    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-sxrn4_13f8df0b-6a12-4dca-95f4-ca467c9aad65/kube-rbac-proxy-main/0.log"
Jul 16 01:21:03.324846 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:21:03.324807    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-sxrn4_13f8df0b-6a12-4dca-95f4-ca467c9aad65/kube-rbac-proxy-self/0.log"
Jul 16 01:21:03.347566 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:21:03.347537    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-sxrn4_13f8df0b-6a12-4dca-95f4-ca467c9aad65/openshift-state-metrics/0.log"
Jul 16 01:21:03.381699 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:21:03.381621    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c888296a-9f5e-47a1-ad37-985053642fc6/prometheus/0.log"
Jul 16 01:21:03.410322 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:21:03.410297    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c888296a-9f5e-47a1-ad37-985053642fc6/config-reloader/0.log"
Jul 16 01:21:03.435674 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:21:03.435648    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c888296a-9f5e-47a1-ad37-985053642fc6/thanos-sidecar/0.log"
Jul 16 01:21:03.461691 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:21:03.461661    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c888296a-9f5e-47a1-ad37-985053642fc6/kube-rbac-proxy-web/0.log"
Jul 16 01:21:03.486412 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:21:03.486371    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c888296a-9f5e-47a1-ad37-985053642fc6/kube-rbac-proxy/0.log"
Jul 16 01:21:03.510563 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:21:03.510536    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c888296a-9f5e-47a1-ad37-985053642fc6/kube-rbac-proxy-thanos/0.log"
Jul 16 01:21:03.534395 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:21:03.534368    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c888296a-9f5e-47a1-ad37-985053642fc6/init-config-reloader/0.log"
Jul 16 01:21:03.736409 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:21:03.736335    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7f7dd99658-8wp4p_15d0414e-398c-40e6-aa30-ede0844470e8/thanos-query/0.log"
Jul 16 01:21:03.759125 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:21:03.759091    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7f7dd99658-8wp4p_15d0414e-398c-40e6-aa30-ede0844470e8/kube-rbac-proxy-web/0.log"
Jul 16 01:21:03.781225 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:21:03.781196    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7f7dd99658-8wp4p_15d0414e-398c-40e6-aa30-ede0844470e8/kube-rbac-proxy/0.log"
Jul 16 01:21:03.806706 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:21:03.806671    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7f7dd99658-8wp4p_15d0414e-398c-40e6-aa30-ede0844470e8/prom-label-proxy/0.log"
Jul 16 01:21:03.835337 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:21:03.835308    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7f7dd99658-8wp4p_15d0414e-398c-40e6-aa30-ede0844470e8/kube-rbac-proxy-rules/0.log"
Jul 16 01:21:03.861372 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:21:03.861338    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7f7dd99658-8wp4p_15d0414e-398c-40e6-aa30-ede0844470e8/kube-rbac-proxy-metrics/0.log"
Jul 16 01:21:04.999933 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:21:04.999898    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-5b5g8/perf-node-gather-daemonset-rv5xg"]
Jul 16 01:21:05.006650 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:21:05.006626    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5b5g8/perf-node-gather-daemonset-rv5xg"
Jul 16 01:21:05.015993 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:21:05.015964    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5b5g8/perf-node-gather-daemonset-rv5xg"]
Jul 16 01:21:05.122414 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:21:05.122380    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2d145e67-859c-49d4-9133-72e23e3bf71e-podres\") pod \"perf-node-gather-daemonset-rv5xg\" (UID: \"2d145e67-859c-49d4-9133-72e23e3bf71e\") " pod="openshift-must-gather-5b5g8/perf-node-gather-daemonset-rv5xg"
Jul 16 01:21:05.122595 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:21:05.122488    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2d145e67-859c-49d4-9133-72e23e3bf71e-lib-modules\") pod \"perf-node-gather-daemonset-rv5xg\" (UID: \"2d145e67-859c-49d4-9133-72e23e3bf71e\") " pod="openshift-must-gather-5b5g8/perf-node-gather-daemonset-rv5xg"
Jul 16 01:21:05.122595 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:21:05.122538    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2d145e67-859c-49d4-9133-72e23e3bf71e-proc\") pod \"perf-node-gather-daemonset-rv5xg\" (UID: \"2d145e67-859c-49d4-9133-72e23e3bf71e\") " pod="openshift-must-gather-5b5g8/perf-node-gather-daemonset-rv5xg"
Jul 16 01:21:05.122722 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:21:05.122614    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4vtn\" (UniqueName: \"kubernetes.io/projected/2d145e67-859c-49d4-9133-72e23e3bf71e-kube-api-access-c4vtn\") pod \"perf-node-gather-daemonset-rv5xg\" (UID: \"2d145e67-859c-49d4-9133-72e23e3bf71e\") " pod="openshift-must-gather-5b5g8/perf-node-gather-daemonset-rv5xg"
Jul 16 01:21:05.122722 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:21:05.122660    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2d145e67-859c-49d4-9133-72e23e3bf71e-sys\") pod \"perf-node-gather-daemonset-rv5xg\" (UID: \"2d145e67-859c-49d4-9133-72e23e3bf71e\") " pod="openshift-must-gather-5b5g8/perf-node-gather-daemonset-rv5xg"
Jul 16 01:21:05.223327 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:21:05.223274    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4vtn\" (UniqueName: \"kubernetes.io/projected/2d145e67-859c-49d4-9133-72e23e3bf71e-kube-api-access-c4vtn\") pod \"perf-node-gather-daemonset-rv5xg\" (UID: \"2d145e67-859c-49d4-9133-72e23e3bf71e\") " pod="openshift-must-gather-5b5g8/perf-node-gather-daemonset-rv5xg"
Jul 16 01:21:05.223524 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:21:05.223357    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2d145e67-859c-49d4-9133-72e23e3bf71e-sys\") pod \"perf-node-gather-daemonset-rv5xg\" (UID: \"2d145e67-859c-49d4-9133-72e23e3bf71e\") " pod="openshift-must-gather-5b5g8/perf-node-gather-daemonset-rv5xg"
Jul 16 01:21:05.223524 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:21:05.223402    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2d145e67-859c-49d4-9133-72e23e3bf71e-podres\") pod \"perf-node-gather-daemonset-rv5xg\" (UID: \"2d145e67-859c-49d4-9133-72e23e3bf71e\") " pod="openshift-must-gather-5b5g8/perf-node-gather-daemonset-rv5xg"
Jul 16 01:21:05.223524 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:21:05.223449    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2d145e67-859c-49d4-9133-72e23e3bf71e-lib-modules\") pod \"perf-node-gather-daemonset-rv5xg\" (UID: \"2d145e67-859c-49d4-9133-72e23e3bf71e\") " pod="openshift-must-gather-5b5g8/perf-node-gather-daemonset-rv5xg"
Jul 16 01:21:05.223524 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:21:05.223501    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2d145e67-859c-49d4-9133-72e23e3bf71e-proc\") pod \"perf-node-gather-daemonset-rv5xg\" (UID: \"2d145e67-859c-49d4-9133-72e23e3bf71e\") " pod="openshift-must-gather-5b5g8/perf-node-gather-daemonset-rv5xg"
Jul 16 01:21:05.223761 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:21:05.223633    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2d145e67-859c-49d4-9133-72e23e3bf71e-proc\") pod \"perf-node-gather-daemonset-rv5xg\" (UID: \"2d145e67-859c-49d4-9133-72e23e3bf71e\") " pod="openshift-must-gather-5b5g8/perf-node-gather-daemonset-rv5xg"
Jul 16 01:21:05.224067 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:21:05.224040    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2d145e67-859c-49d4-9133-72e23e3bf71e-sys\") pod \"perf-node-gather-daemonset-rv5xg\" (UID: \"2d145e67-859c-49d4-9133-72e23e3bf71e\") " pod="openshift-must-gather-5b5g8/perf-node-gather-daemonset-rv5xg"
Jul 16 01:21:05.224184 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:21:05.224163    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2d145e67-859c-49d4-9133-72e23e3bf71e-podres\") pod \"perf-node-gather-daemonset-rv5xg\" (UID: \"2d145e67-859c-49d4-9133-72e23e3bf71e\") " pod="openshift-must-gather-5b5g8/perf-node-gather-daemonset-rv5xg"
Jul 16 01:21:05.224274 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:21:05.224257    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2d145e67-859c-49d4-9133-72e23e3bf71e-lib-modules\") pod \"perf-node-gather-daemonset-rv5xg\" (UID: \"2d145e67-859c-49d4-9133-72e23e3bf71e\") " pod="openshift-must-gather-5b5g8/perf-node-gather-daemonset-rv5xg"
Jul 16 01:21:05.233928 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:21:05.233897    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4vtn\" (UniqueName: \"kubernetes.io/projected/2d145e67-859c-49d4-9133-72e23e3bf71e-kube-api-access-c4vtn\") pod \"perf-node-gather-daemonset-rv5xg\" (UID: \"2d145e67-859c-49d4-9133-72e23e3bf71e\") " pod="openshift-must-gather-5b5g8/perf-node-gather-daemonset-rv5xg"
Jul 16 01:21:05.323205 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:21:05.323178    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5b5g8/perf-node-gather-daemonset-rv5xg"
Jul 16 01:21:05.509235 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:21:05.509058    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5b5g8/perf-node-gather-daemonset-rv5xg"]
Jul 16 01:21:05.515920 ip-10-0-143-65 kubenswrapper[2563]: W0716 01:21:05.513695    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod2d145e67_859c_49d4_9133_72e23e3bf71e.slice/crio-66d19b8c09f5a05a9e8ac8f21d30728b1be32e4cdb49954fa7b79d339363f9f1 WatchSource:0}: Error finding container 66d19b8c09f5a05a9e8ac8f21d30728b1be32e4cdb49954fa7b79d339363f9f1: Status 404 returned error can't find the container with id 66d19b8c09f5a05a9e8ac8f21d30728b1be32e4cdb49954fa7b79d339363f9f1
Jul 16 01:21:06.232637 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:21:06.232589    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5b5g8/perf-node-gather-daemonset-rv5xg" event={"ID":"2d145e67-859c-49d4-9133-72e23e3bf71e","Type":"ContainerStarted","Data":"0934ce017fd6f845ebec60e3fc872f32879ade62779ebd000c1950b5d1a103a0"}
Jul 16 01:21:06.233162 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:21:06.232663    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5b5g8/perf-node-gather-daemonset-rv5xg" event={"ID":"2d145e67-859c-49d4-9133-72e23e3bf71e","Type":"ContainerStarted","Data":"66d19b8c09f5a05a9e8ac8f21d30728b1be32e4cdb49954fa7b79d339363f9f1"}
Jul 16 01:21:06.233472 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:21:06.233445    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-5b5g8/perf-node-gather-daemonset-rv5xg"
Jul 16 01:21:06.252322 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:21:06.252265    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-5b5g8/perf-node-gather-daemonset-rv5xg" podStartSLOduration=2.252244 podStartE2EDuration="2.252244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 01:21:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 01:21:06.249643821 +0000 UTC m=+2438.684905561" watchObservedRunningTime="2026-07-16 01:21:06.252244 +0000 UTC m=+2438.687505739"
Jul 16 01:21:07.848358 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:21:07.848326    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-9tmlj_d75c83b3-ea33-4e0a-b726-6ea607f587a7/dns/0.log"
Jul 16 01:21:07.874050 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:21:07.874020    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-9tmlj_d75c83b3-ea33-4e0a-b726-6ea607f587a7/kube-rbac-proxy/0.log"
Jul 16 01:21:08.193062 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:21:08.192991    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-gzbcl_bd81358e-a649-4ba2-b63e-d9ba85ee93db/dns-node-resolver/0.log"
Jul 16 01:21:08.780853 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:21:08.780799    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-5fd576b4b9-pwb8z_4562419f-a9c7-499b-b9c3-c1d4ccad25d1/registry/0.log"
Jul 16 01:21:08.829424 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:21:08.829398    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-rk85k_f0b6d16f-9a13-480c-b981-6481983dbd91/node-ca/0.log"
Jul 16 01:21:09.960676 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:21:09.960645    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7ffc9f7c5d-k7f7b_b98801ba-36e7-4e48-84b8-56709b911b43/kube-auth-proxy/0.log"
Jul 16 01:21:10.889514 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:21:10.889475    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-xnkzv_4c6aeea8-fa7c-44db-89df-7bb67070d188/serve-healthcheck-canary/0.log"
Jul 16 01:21:11.391729 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:21:11.391698    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-57rzd_908a59b2-e065-4d36-a3a2-8de114feaf35/kube-rbac-proxy/0.log"
Jul 16 01:21:11.414202 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:21:11.414171    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-57rzd_908a59b2-e065-4d36-a3a2-8de114feaf35/exporter/0.log"
Jul 16 01:21:11.438493 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:21:11.438455    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-57rzd_908a59b2-e065-4d36-a3a2-8de114feaf35/extractor/0.log"
Jul 16 01:21:13.256713 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:21:13.256686    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-5b5g8/perf-node-gather-daemonset-rv5xg"
Jul 16 01:21:13.678741 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:21:13.678716    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-cgjd4_f670d048-05a1-44e0-bb47-f33afeba3880/manager/0.log"
Jul 16 01:21:14.312017 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:21:14.311976    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5cf8c55d45-8d6hh_c9446ea3-dac9-4a1a-9ad3-1fda85d9f151/manager/0.log"
Jul 16 01:21:14.331269 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:21:14.331236    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5cf8c55d45-8d6hh_c9446ea3-dac9-4a1a-9ad3-1fda85d9f151/copy-manifests/0.log"
Jul 16 01:21:15.769327 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:21:15.769299    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-c857fdb5b-5bxrx_70e026c5-9607-43bd-b482-8555fb4f6c41/manager/0.log"
Jul 16 01:21:15.842758 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:21:15.842719    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-q58xz_8b4ddb9a-aabe-424a-a646-73c81269deb7/openshift-lws-operator/0.log"
Jul 16 01:21:16.163990 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:21:16.163958    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7ffc9f7c5d-k7f7b_b98801ba-36e7-4e48-84b8-56709b911b43/kube-auth-proxy/0.log"
Jul 16 01:21:17.621179 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:21:17.621143    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7ffc9f7c5d-k7f7b_b98801ba-36e7-4e48-84b8-56709b911b43/kube-auth-proxy/0.log"
Jul 16 01:21:22.597124 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:21:22.597091    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-l97c7_49429502-4592-4b95-a41b-7f831eff48da/kube-multus-additional-cni-plugins/0.log"
Jul 16 01:21:22.619944 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:21:22.619909    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-l97c7_49429502-4592-4b95-a41b-7f831eff48da/egress-router-binary-copy/0.log"
Jul 16 01:21:22.643439 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:21:22.643408    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-l97c7_49429502-4592-4b95-a41b-7f831eff48da/cni-plugins/0.log"
Jul 16 01:21:22.668273 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:21:22.668246    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-l97c7_49429502-4592-4b95-a41b-7f831eff48da/bond-cni-plugin/0.log"
Jul 16 01:21:22.692903 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:21:22.692874    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-l97c7_49429502-4592-4b95-a41b-7f831eff48da/routeoverride-cni/0.log"
Jul 16 01:21:22.716799 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:21:22.716776    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-l97c7_49429502-4592-4b95-a41b-7f831eff48da/whereabouts-cni-bincopy/0.log"
Jul 16 01:21:22.739181 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:21:22.739155    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-l97c7_49429502-4592-4b95-a41b-7f831eff48da/whereabouts-cni/0.log"
Jul 16 01:21:22.948112 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:21:22.948045    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-h4ks8_090eddb9-3f4c-4a94-9b62-4f6bf8f38428/kube-multus/0.log"
Jul 16 01:21:23.097377 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:21:23.097349    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-bkf2h_051e94d1-76fa-484b-b118-b29951f4c5ae/network-metrics-daemon/0.log"
Jul 16 01:21:23.119005 ip-10-0-143-65 kubenswrapper[2563]: I0716 01:21:23.118984    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-bkf2h_051e94d1-76fa-484b-b118-b29951f4c5ae/kube-rbac-proxy/0.log"